LINUX.ORG.RU
ФорумAdmin

haproxy не видит patroni

 ,


0

1

Здравствуйте, коллеги!

на текущий момент настроена кластерная связка на 3 ноды: etcd, postgresql12-server, patroni и весь этот зоопарк нормально работает.

За основу был взят мануал https://itdraft.ru/2023/08/14/nastrojka-otkazoustojchivogo-klastera-postgresql-v-linux/

pgbouncer мне пока не нужен.

Установил haproxy, адаптировал конфиг под Altlinux и не выходит каменный цветок.

systemctl status haproxy
...
Jun 27 15:27:10 host-101 haproxy[9409]: [NOTICE]   (9409) : New worker (9411) forked
Jun 27 15:27:10 host-101 haproxy[9409]: [NOTICE]   (9409) : Loading success.
Jun 27 15:27:10 host-101 systemd[1]: Started HAProxy Load Balancer.
Jun 27 15:27:10 host-101 haproxy[9411]: [WARNING]  (9411) : Server postgres_master/host-101 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 4ms>
Jun 27 15:27:10 host-101 haproxy[9411]: [ALERT]    (9411) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
Jun 27 15:27:10 host-101 haproxy[9411]: [WARNING]  (9411) : Server postgres_replicas/host-102 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1>
Jun 27 15:27:10 host-101 haproxy[9411]: [WARNING]  (9411) : Server postgres_replicas_sync/host-101 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check durati>
Jun 27 15:27:10 host-101 haproxy[9411]: [WARNING]  (9411) : Server postgres_replicas_sync/host-102 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check durati>
Jun 27 15:27:10 host-101 haproxy[9411]: [ALERT]    (9411) : proxy 'postgres_replicas_sync' has no server available!
Jun 27 15:27:10 host-101 haproxy[9411]: [WARNING]  (9411) : Server postgres_replicas_async/host-102 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check dura

вот отрывок из

cat /etc/haproxy/haproxy.cfg 
global
    maxconn 100000
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /var/lib/haproxy/stats mode 660 level admin expose-fd listeners
    pidfile /run/haproxy.pid
    stats timeout 30s
    user _haproxy
    group _haproxy
    daemon

defaults
    mode               tcp
    log                global
    retries            2
    timeout queue      5s
    timeout connect    5s
    timeout client     60m
    timeout server     60m
    timeout check      15s

listen stats
    mode http
    bind 192.168.12.101:7000
    stats enable
    stats uri /

listen postgres_master
    bind *:5000
    maxconn 10000
    option tcplog
    option httpchk OPTIONS /master HTTP/1.1
    http-check expect status 200
    default-server inter 3s fastinter 1s fall 3 rise 4 on-marked-down shutdown-sessions
    server host-101 192.168.12.101:5432 check port 8008
    server host-102 192.168.12.102:5432 check port 8008
    

listen postgres_replicas
    bind *:5001
    maxconn 10000
    option tcplog
    option httpchk OPTIONS /replica HTTP/1.1
    balance roundrobin
    http-check expect status 200
    default-server inter 3s fastinter 1s fall 3 rise 2 on-marked-down shutdown-sessions
    server host-101 192.168.12.101:5432 check port 8008
    server host-102 192.168.12.102:5432 check port 8008

8008 порт отдает patroni и он нормально работает

patronictl -c /etc/patroni/patroni.yml list pg_cluster
+ Cluster: pg_cluster ------+---------+---------+----+-----------+
| Member   | Host           | Role    | State   | TL | Lag in MB |
+----------+----------------+---------+---------+----+-----------+
| host-101 | 192.168.12.101 | Replica | running |  3 |         0 |
| host-102 | 192.168.12.102 | Leader  | running |  3 |           |
+----------+----------------+---------+---------+----+-----------+

Я полагаю, что haproxy что-то не то спрашивает у patroni и они друг друга не понимают.

Как, вообще, через telenet поопрашивать patroni?

И что тут может быть не так?


Ответ на: комментарий от vadv

попробуй курлануть 192.168.12.10{1,2}:8008/cluster - вангую что тупо patroni не слушает.

#curl 192.168.12.101:8008/
{"state": "running", "postmaster_start_time": "2024-06-27 15:11:17.078799+03:00", "role": "replica", "server_version": 120019, "xlog": {"received_location": 50332776, "replayed_location": 50332776, "replayed_timestamp": null, "paused": false}, "timeline": 3, "dcs_last_seen": 1719497827, "database_system_identifier": "7385148893721311398", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}

#curl 192.168.12.102:8008/
{"state": "running", "postmaster_start_time": "2024-06-27 15:09:28.828499+03:00", "role": "master", "server_version": 120019, "xlog": {"location": 50332776}, "timeline": 3, "replication": [{"usename": "patroni_replica", "application_name": "host-101", "client_addr": "192.168.12.101", "state": "streaming", "sync_state": "async", "sync_priority": 0}], "dcs_last_seen": 1719497906, "database_system_identifier": "7385148893721311398", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}

Эти запросы были выполнены с 192.168.12.101

HighMan
() автор топика
Последнее исправление: HighMan (всего исправлений: 1)
Ответ на: комментарий от vadv

а какой http code на /master ?

Я, так понимаю, HTTP/1.0

Это tcpdump слушает на 102й 101ую

E..4..@.@......e...f...H...a........iY..............
17:52:34.385756 IP host-101.36866 > host-102.http-alt: Flags [P.], seq 3003059810:3003059835, ack 2488388218, win 502, length 25
E..A..@.@......e...f...H...b.Q.zP...
...GET /replica HTTP/1.0
HighMan
() автор топика
Ответ на: комментарий от vadv

http code 200 или 500?

503

Jun 27 18:07:34 host-101 haproxy[36218]: [WARNING]  (36218) : Server postgres_master/host-101 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 4>
Jun 27 18:07:34 host-101 haproxy[36218]: [ALERT]    (36218) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
Jun 27 18:07:34 host-101 haproxy[36218]: [WARNING]  (36218) : Server postgres_replicas/host-102 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration:>
Jun 27 18:07:34 host-101 haproxy[36218]: [WARNING]  (36218) : Server postgres_replicas_sync/host-101 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check dura>
Jun 27 18:07:35 host-101 haproxy[36218]: [WARNING]  (36218) : Server postgres_replicas_sync/host-102 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check dura>
Jun 27 18:07:35 host-101 haproxy[36218]: [ALERT]    (36218) : proxy 'postgres_replicas_sync' has no server available!
Jun 27 18:07:35 host-101 haproxy[36218]: [WARNING]  (36218) : Server postgres_replicas_async/host-102 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check dur>
HighMan
() автор топика
Ответ на: комментарий от cobold

Это запрос, а нужен ответ. curl -v запусти

Я выше приводил отдачу curl

# curl 192.168.12.102:8008/master
{"state": "running", "postmaster_start_time": "2024-06-27 15:09:28.828499+03:00", "role": "master", "server_version": 120019, "xlog": {"location": 50332776}, "timeline": 3, "replication": [{"usename": "patroni_replica", "application_name": "host-101", "client_addr": "192.168.12.101", "state": "streaming", "sync_state": "async", "sync_priority": 0}], "dcs_last_seen": 1719558866, "database_system_identifier": "7385148893721311398", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}

# curl 192.168.12.102:8008/sync
{"state": "running", "postmaster_start_time": "2024-06-27 15:09:28.828499+03:00", "role": "master", "server_version": 120019, "xlog": {"location": 50332776}, "timeline": 3, "replication": [{"usename": "patroni_replica", "application_name": "host-101", "client_addr": "192.168.12.101", "state": "streaming", "sync_state": "async", "sync_priority": 0}], "dcs_last_seen": 1719558916, "database_system_identifier": "7385148893721311398", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}

Ну и т.д.

Это я со 101й машины курлю 102ую. 101 и 102 настроены одинаково: etcd, postgresql, patroni …

Если со 101 курлить себя же, то вывод аналогичен, полученному со 102ой.

Ну и из другой подсети, с моей рабочей машины, все курлится без вопросов. Вот я и не могу понять, чего не хватает haproxy

Каким образм haproxy получает ошибку 503???

HighMan
() автор топика
Последнее исправление: HighMan (всего исправлений: 2)
Ответ на: комментарий от BOOBLIK

Ты бы ещё strace’ом http response искал. Было бы эпичнее.

Искал чем привычнее.

Я даже до curl «додумался» после подсказки :) Мне ping, telnet и tcpdump привычнее.

HighMan
() автор топика
Ответ на: комментарий от cobold

Взял и тупо скопировал конфиг haproxy из https://habr.com/ru/articles/482314/

Он простой как 3 копейки. Вставил вместо своего. И все то же самое.

# haproxy -f /etc/haproxy/haproxy.cfg
[WARNING]  (203474) : Server postgres/postgresql1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Может haproxy в дистре альта кривой???

Не-а. Поставил на свою Fedora haproxy и результат ровно тот же. Ашыпка 503.

Что, за бред???

Со своей же машины, где только что спасовал haproxy:

$ curl 192.168.12.101:8008
{"state": "running", "postmaster_start_time": "2024-06-27 15:11:17.078799+03:00", "role": "replica", "server_version": 120019, "xlog": {"received_location": 50332776, "replayed_location": 50332776, "replayed_timestamp": null, "paused": false}, "timeline": 3, "dcs_last_seen": 1719563197, "database_system_identifier": "7385148893721311398", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}
HighMan
() автор топика
Последнее исправление: HighMan (всего исправлений: 2)
Ответ на: комментарий от vadv

на replica - должен быть 500, на master (192.168.12.102) - http code 200

Я торможу!

The following requests to Patroni REST API will return HTTP status code 200 only when the Patroni node is running as the primary with leader lock:

Это из документации patroni/rest-api

Т.е. лишь leader вертает 200. Все остальные что-то иное. Скорее всего 503.

Вот, блин, моя невнимательность!

# patronictl -c /etc/patroni/patroni.yml list pg_cluster
+ Cluster: pg_cluster ------+--------------+---------+----+-----------+
| Member   | Host           | Role         | State   | TL | Lag in MB |
+----------+----------------+--------------+---------+----+-----------+
| host-101 | 192.168.12.101 | Leader       | running |  4 |           |
| host-102 | 192.168.12.102 | Sync Standby | running |  4 |         0 |
+----------+----------------+--------------+---------+----+-----------+

# curl -v 192.168.12.102:8008/primary
*   Trying 192.168.12.102:8008...
* Connected to 192.168.12.102 (192.168.12.102) port 8008
> GET /primary HTTP/1.1
> Host: 192.168.12.102:8008
> User-Agent: curl/8.6.0
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
< Server: BaseHTTP/0.6 Python/3.9.18
< Date: Fri, 28 Jun 2024 10:03:49 GMT
< Content-Type: application/json
< 
* Closing connection
{"state": "running", "postmaster_start_time": "2024-06-28 12:49:59.687670+03:00", "role": "replica", "server_version": 120019, "xlog": {"received_location": 134217920, "replayed_location": 134217920, "replayed_timestamp": "2024-06-28 12:50:11.769533+03:00", "paused": false}, "sync_standby": true, "timeline": 4, "dcs_last_seen": 1719569021, "database_system_identifier": "7385484064715830130", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}

# curl -v 192.168.12.101:8008/primary
*   Trying 192.168.12.101:8008...
* Connected to 192.168.12.101 (192.168.12.101) port 8008
> GET /primary HTTP/1.1
> Host: 192.168.12.101:8008
> User-Agent: curl/8.6.0
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BaseHTTP/0.6 Python/3.9.18
< Date: Fri, 28 Jun 2024 10:04:17 GMT
< Content-Type: application/json
< 
* Closing connection
{"state": "running", "postmaster_start_time": "2024-06-28 12:43:20.152746+03:00", "role": "master", "server_version": 120019, "xlog": {"location": 134217920}, "timeline": 4, "replication": [{"usename": "replicator", "application_name": "host-102", "client_addr": "192.168.12.102", "state": "streaming", "sync_state": "sync", "sync_priority": 1}], "dcs_last_seen": 1719569051, "database_system_identifier": "7385484064715830130", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}

Похоже, что haproxy заработал

HighMan
() автор топика
Ответ на: комментарий от vadv

Не. Ни фига проблема не решилась.

# patronictl -c /etc/patroni/patroni.yml list pg_cluster
+ Cluster: pg_cluster ------+--------------+---------+----+-----------+
| Member   | Host           | Role         | State   | TL | Lag in MB |
+----------+----------------+--------------+---------+----+-----------+
| host-100 | 192.168.12.100 | Leader       | running |  5 |           |
| host-101 | 192.168.12.101 | Sync Standby | running |  5 |         0 |
| host-102 | 192.168.12.102 | Replica      | running |  5 |         0 |
+----------+----------------+--------------+---------+----+-----------+

# systemctl status haproxy.service
....
Jul 01 10:27:03 host-100 haproxy[1568265]: [WARNING]  (1568265) : Server postgres_master/host-100 is DOWN, reason: Layer7 invalid response, check duration: 2ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 01 10:27:03 host-100 haproxy[1568265]: [ALERT]    (1568265) : sendmsg()/writev() failed in logger #1: No such file or directory (errno=2)
Jul 01 10:27:03 host-100 haproxy[1568265]: [WARNING]  (1568265) : Server postgres_master/host-101 is DOWN, reason: Layer7 invalid response, check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 01 10:27:03 host-100 haproxy[1568265]: [WARNING]  (1568265) : Server postgres_master/host-102 is DOWN, reason: Layer7 invalid response, check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 01 10:27:03 host-100 haproxy[1568265]: [ALERT]    (1568265) : proxy 'postgres_master' has no server available!
Jul 01 10:27:03 host-100 haproxy[1568265]: [WARNING]  (1568265) : Server postgres_replicas/host-100 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 10ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 01 10:27:04 host-100 haproxy[1568265]: [WARNING]  (1568265) : Server postgres_replicas_sync/host-100 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 6ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 01 10:27:04 host-100 haproxy[1568265]: [WARNING]  (1568265) : Server postgres_replicas_sync/host-102 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 12ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 01 10:27:04 host-100 haproxy[1568265]: [WARNING]  (1568265) : Server postgres_replicas_async/host-100 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 7ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 01 10:27:04 host-100 haproxy[1568265]: [WARNING]  (1568265) : Server postgres_replicas_async/host-101 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 11ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Как видно из вывода patronictl лидером является host-100, но шибанутый haproxy все равно выдает код 503.

Откуда он его берет - не ведомо.

# curl 192.168.12.100:8008/master -v
*   Trying 192.168.12.100:8008...
* Connected to 192.168.12.100 (192.168.12.100) port 8008
> GET /master HTTP/1.1
> Host: 192.168.12.100:8008
> User-Agent: curl/8.6.0
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BaseHTTP/0.6 Python/3.9.18
< Date: Mon, 01 Jul 2024 07:32:38 GMT
< Content-Type: application/json
< 
* Closing connection
{"state": "running", "postmaster_start_time": "2024-06-28 17:24:29.485522+03:00", "role": "master", "server_version": 120019, "xlog": {"location": 721420288}, "timeline": 5, "replication": [{"usename": "patroni_replica", "application_name": "host-101", "client_addr": "192.168.12.101", "state": "streaming", "sync_state": "sync", "sync_priority": 1}, {"usename": "patroni_replica", "application_name": "host-102", "client_addr": "192.168.12.102", "state": "streaming", "sync_state": "async", "sync_priority": 0}], "dcs_last_seen": 1719819151, "database_system_identifier": "7385553500470229262", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}

Код 200!

# curl 192.168.12.101:8008/master -v
*   Trying 192.168.12.101:8008...
* Connected to 192.168.12.101 (192.168.12.101) port 8008
> GET /master HTTP/1.1
> Host: 192.168.12.101:8008
> User-Agent: curl/8.6.0
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
< Server: BaseHTTP/0.6 Python/3.9.18
< Date: Mon, 01 Jul 2024 07:33:48 GMT
< Content-Type: application/json
< 
* Closing connection
{"state": "running", "postmaster_start_time": "2024-06-28 17:23:36.056707+03:00", "role": "replica", "server_version": 120019, "xlog": {"received_location": 721420288, "replayed_location": 721420288, "replayed_timestamp": "2024-06-28 17:44:52.823814+03:00", "paused": false}, "sync_standby": true, "timeline": 5, "dcs_last_seen": 1719819221, "database_system_identifier": "7385553500470229262", "patroni": {"version": "3.0.2", "scope": "pg_cluster"}}

Вот тут честный код возврата 503, абы host-101 не является в данный момент лидером.

Если же курлануть curl 192.168.12.101:8008/sync -v, то код возврата будет 200, так как host-101(192.168.12.101) является Sync Standby

HighMan
() автор топика
Последнее исправление: HighMan (всего исправлений: 2)