FATAL: Ident authentication failed for user "netbox"

[kjanezic95]

Hello, I have been trying to move netbox from an old CentOS Stream 8 server to a RHEL 9. The version that I want to move is 2.11.11 (I had issues upgrading it on the old server because of ancient postgresql versions, so I want to move it and upgrade it there).

As per the instructions (https://netboxlabs.com/docs/netbox/en/stable/installation/, https://netboxlabs.com/docs/netbox/en/stable/administration/replicating-netbox/), I have set up a new instance and imported the database. Import goes fine, the user is set up with the correct password.

CREATE DATABASE netbox;
CREATE USER netbox WITH PASSWORD '<SUPER_SECURE_PASS>';
ALTER DATABASE netbox OWNER TO netbox;
sudo -u postgres psql netbox < netbox-v10.sql  #v10 refers to the postgresql version on the old server

But after runing ./upgrade.sh and accessing the server,

 Server Error
There was a problem with your request. Please contact an administrator.
The complete exception is provided below:
<class 'django.db.utils.OperationalError'>
FATAL:  Ident authentication failed for user "netbox"
Python version: 3.6.8  # This is weird, because on the new server the version is 3.9.18. The old version is 3.6.8.
NetBox version: 2.11.11

CMD login to DB works fine.

[root@netbox2 opt]# psql --username netbox --password --host localhost netbox
Password:
psql (13.16)
Type "help" for help.
netbox=> \conninfo
You are connected to database "netbox" as user "netbox" on host "localhost" (address "::1") at port "5432".

I do not know what else I can do to fix this issue. Any advice will be greatly apreciated.
Thank you

---

Logs, configs, etc...:

./uprade.sh works fine (or at least I dont see any errors).

[root@netbox2 opt]# dnf list --installed postgresql-server
Updating Subscription Management repositories.
Installed Packages
postgresql-server.x86_64                                                                                  13.16-1.el9_4
[root@netbox2 opt]# python3 -V
Python 3.9.18

Configuration file:

ALLOWED_HOSTS = ['netbox.<DOMAIN>.com', '<SERVER_IP>', '127.0.0.1']
# PostgreSQL database configuration. See the Django documentation for a complete list of available parameters:
#   https://docs.djangoproject.com/en/stable/ref/settings/#databases
DATABASE = {
    'NAME': 'netbox',         # Database name
    'USER': 'netbox',               # PostgreSQL username
    'PASSWORD': '<SUPER_SECURE_PASS>',           # PostgreSQL password
    'HOST': 'localhost',      # Database server
    'PORT': '',               # Database port (leave blank for default)
    'CONN_MAX_AGE': 300,      # Max database connection age
}

Config copy went something like this (this was taken from the 2.9.0 to 2.11.11 upgrade, hence the oldver & newver):

oldver="2.11.11"
newver="2.11.11"
cp OLD.netbox-$oldver/local_requirements.txt netbox-$newver/
cp OLD.netbox-$oldver/netbox/netbox/configuration.py netbox-$newver/netbox/netbox/
cp OLD.netbox-$oldver/netbox/netbox/ldap_config.py netbox-$newver/netbox/netbox/
cp -r OLD.netbox-$oldver/netbox/media/ netbox-$newver/netbox/
cp -r OLD.netbox-$oldver/netbox/{scripts,reports}/ netbox-$newver/netbox/
cp OLD.netbox-$oldver/gunicorn_config.py netbox-$newver/
cp /opt/netbox/contrib/gunicorn.py /opt/netbox/.

PostgreSQL is listening.

[root@netbox2 netbox]# netstat -peanut | grep -i 5432
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      26         267376     71053/postmaster
tcp6       0      0 ::1:5432                :::*                    LISTEN      26         267375     71053/postmaster

PostgreSQL log is empty, it does not show any connection attempts. Todays log (/var/lib/pgsql/data/log/postgresql-Mon.log) is empty. After restarting postgresql service, it shows normal entries.

[root@netbox2 log]# tail -f postgresql-Mon.log
2024-09-09 06:58:05.853 CEST [71053] LOG:  received fast shutdown request
2024-09-09 06:58:05.855 CEST [71053] LOG:  aborting any active transactions
2024-09-09 06:58:05.860 CEST [71053] LOG:  background worker "logical replication launcher" (PID 71061) exited with exit code 1
2024-09-09 06:58:05.861 CEST [71056] LOG:  shutting down
2024-09-09 06:58:05.873 CEST [71053] LOG:  database system is shut down
2024-09-09 06:58:05.936 CEST [119934] LOG:  starting PostgreSQL 13.16 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3), 64-bit
2024-09-09 06:58:05.936 CEST [119934] LOG:  listening on IPv6 address "::1", port 5432
2024-09-09 06:58:05.936 CEST [119934] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2024-09-09 06:58:05.938 CEST [119934] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-09-09 06:58:05.939 CEST [119934] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2024-09-09 06:58:05.943 CEST [119936] LOG:  database system was shut down at 2024-09-09 06:58:05 CEST
2024-09-09 06:58:05.951 CEST [119934] LOG:  database system is ready to accept connections

Some changes I did to postgresql to allow connections from CMD (/var/lib/pgsql/data/pg_hba.conf):

# TYPE  DATABASE        USER            ADDRESS                 METHOD
# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5         #<--- CHANGE
#host    all             all             127.0.0.1/32            ident
# IPv6 local connections:
host    all             all             ::1/128                 md5         #<--- CHANGE
#host    all             all             ::1/128                 ident
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     peer
host    replication     all             127.0.0.1/32            ident
host    replication     all             ::1/128                 ident

FW is shut down .

[root@netbox2 netbox]# firewall-cmd --state
not running

SELinux shows no errors (grep AVC /var/log/audit/audit.log -> no results).

[candlerb]

I have no idea why the "wrong" python version is being reported.

However, I will point out that you don't need to install 2.11.11 on the new server. You can install 3.7.8, which is the final version that can import a database from 2.11.x.

This gives the following plan:

First: install 3.7.8 on the new server (with an empty database), and test that it's working properly. No exceptions etc. You can create test sites and devices in this database to check.

Second: drop the database, import the database dump from 2.11.11, and run ./upgrade.sh. That will modify the database to work with 3.7.8. Again, check that it's all working, but this time it will have the imported data.

Third: upgrade this system to the final version you want, e.g. v4.0.11 or v4.1.0

Doing it this way avoids the problems from installing an ancient Netbox on a modern system, especially broken dependencies. This is because Netbox installs are not reproducible; it doesn't define the exact or "frozen" full sent of dependencies, only a set of top-level constraints, and often new versions of packages are released which break those dependencies. If you must install v2.11.11 then there's a repo of netbox "freezes" out there somewhere. What I'm trying to say though is, because you're not trying to import your database from something earlier than 2.11, you can jump straight to 3.7

[kjanezic95]

Hello, thank you very much for the reply and the explanation. Doing it this ways yields better results, but at the ./upgrade.sh the process stops at DB tasks. I can see that this ("DELETE FROM "extras_objectchange" WHERE "extras_objectchange"."id" IN (195, 194)") is the issue, but I do not know how to tackle it. Maybe I could add a dummy entry to be deleted?

What I did:

# Download netbox v3.7.8 and set up configs
# Test that the netbox is accesible with empty DB, test LDAP login
systemctl stop netbox netbox-rq
sudo -u postgres psql
# DROP DATABASE netbox;
# CREATE DATABASE netbox;
# ALTER DATABASE netbox OWNER TO netbox;
sudo -u postgres psql netbox < netbox-v10.sql # No visible errors, or errors in postgresql log
./upgrade.sh

./upgrade.sh error:

Running migrations:
  Applying tenancy.0002_tenant_ordering... OK
  Applying extras.0060_customlink_button_class... OK
  Applying extras.0061_extras_change_logging... OK
  Applying extras.0062_clear_secrets_changelog...Skipping config initialization (database unavailable)
Traceback (most recent call last):
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox-3.7.8/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/netbox-3.7.8/netbox/extras/migrations/0062_clear_secrets_changelog.py", line 12, in clear_secrets_changelog
    ObjectChange.objects.filter(changed_object_type__in=content_type_ids).delete()
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/models/query.py", line 1148, in delete
    deleted, _rows_count = collector.delete()
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/models/deletion.py", line 506, in delete
    count = query.delete_batch(pk_list, self.using)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/models/sql/subqueries.py", line 42, in delete_batch
    num_deleted += self.do_query(
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/models/sql/subqueries.py", line 20, in do_query
    cursor = self.get_compiler(using).execute_sql(CURSOR)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
    cursor.execute(sql, params)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/netbox-3.7.8/venv/lib64/python3.9/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.InternalError: current transaction is aborted, commands ignored until end of transaction block

DB error:

2024-09-09 10:07:29.974 CEST [124051] ERROR:  relation "core_configrevision" does not exist at character 136
2024-09-09 10:07:29.974 CEST [124051] STATEMENT:  SELECT "core_configrevision"."id", "core_configrevision"."created", "core_configrevision"."comment", "core_configrevision"."data" FROM "core_configrevision" ORDER BY "core_configrevision"."created" ASC LIMIT 1
2024-09-09 10:07:29.976 CEST [124051] ERROR:  current transaction is aborted, commands ignored until end of transaction block
2024-09-09 10:07:29.976 CEST [124051] STATEMENT:  DELETE FROM "extras_objectchange" WHERE "extras_objectchange"."id" IN (195, 194)

[candlerb]

I can see that this ("DELETE FROM "extras_objectchange" WHERE "extras_objectchange"."id" IN (195, 194)") is the issue

No, the issue is the previous statement (SELECT ... FROM "core_configrevision") failed. Once you have an one error in a transaction, all subsequent statements are automatically failed ("current transaction is aborted, commands ignored until end of transaction block")

There's a current known problem with dependencies in migrations, see #17401 and #17379 (comment)

People have been reporting the issue with extras_objectchange, but core_configrevision may be another table with dependency problems.

[kjanezic95]

Thank you for the explanation. After reading these threads I tried simply removing this line from file /opt/netbox/netbox/extras/migrations/0062_clear_secrets_changelog.py

Error:

  File "/opt/netbox-3.7.8/netbox/extras/migrations/0062_clear_secrets_changelog.py", line 12, in clear_secrets_changelog
    ObjectChange.objects.filter(changed_object_type__in=content_type_ids).delete()

Comented line:
#ObjectChange.objects.filter(changed_object_type__in=content_type_ids).delete()

After that ./upgrade.sh was successful, except this issue:

Running migrations:
  Applying dcim.0145_site_remove_deprecated_fields...Traceback (most recent call last):
  ....
    File "/opt/netbox-3.7.8/netbox/dcim/migrations/0145_site_remove_deprecated_fields.py", line 15, in check_legacy_data
  raise DataError(
django.db.utils.DataError: Unable to proceed with deleting asn field from Site model: Found 2 sites with legacy ASN data. 
Please ensure all legacy site ASN data has been migrated to ASN objects before proceeding. Or, set the 
NETBOX_DELETE_LEGACY_DATA environment variable to bypass this safeguard and delete all legacy site ASN data.

Beeing unfamiliar with ASN data, determined it with Google as "Autonomous System Number (ASN) is a numeric identifier used in the BGP protocol to identify which autonomous system a particular prefix is originating and transiting through.". I am guessing this should be some entry in the devices doing BGP? Didn't find anything in old server entries so I set up the variable and allowed it to delete the data. Comparing old and new server entries shows no differences, so I'm hoping everything is OK. If I was mistaken, where can I find this data? We only use a couple of devices with BGP so it shouldn't be hard to copy to new server.

export NETBOX_DELETE_LEGACY_DATA="true"
./upgrade.sh

After this, the upgrade from v2.11.11 to v3.7.8 was successful.

---

Immediately after this I tried upgrade to v4.1.0. After building and installing newer python of course..

There was one error, which was fixed as per this comment:

...
  File "/opt/netbox-4.1.0/netbox/users/migrations/0005_alter_user_table.py", line 17, in update_content_types
    CustomField.objects.filter(related_object_type_id=netboxuser_ct.id).update(related_object_type_id=user_ct.id)
    raise FieldError(
django.core.exceptions.FieldError: Cannot resolve keyword 'related_object_type_id' into field. Choices are: choice_se
t, choice_set_id, created, default, description, filter_logic, group_name, id, is_cloneable, label, last_updated, nam
e, object_type, object_type_id, object_types, required, search_weight, type, ui_editable, ui_visible, validation_maxi
mum, validation_minimum, validation_regex, weight

After this the upgrade from v3.7.8 to v4.1.0 was successful.

Thank you again for the help.