| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-31 | |||
| 12:57:14 | sean-k-mooney | aspiers: also did you add oslo_log support to lnav as a plugin if so i might have to try that | |
| 12:57:31 | aspiers | sean-k-mooney: that's what the link above is | |
| 12:57:46 | aspiers | sean-k-mooney: it gets installed via "lnav -i extra" | |
| 12:57:56 | aspiers | just type that | |
| 12:58:02 | sean-k-mooney | ya i clicked and got confused with your core dump comment but imm looking at teh paste now | |
| 13:03:00 | sean-k-mooney | hum well i cant really tell what cased that seg fault other thin it appers to be a result of a getattr call on a randomp python object. so ill blame eventlest by default if you can repoduce with the pytho debuger instad of gdb that might help | |
| 13:05:00 | sean-k-mooney | aspiers: can you try running the conductor with pdb | |
| 13:05:03 | aspiers | sean-k-mooney: I'm trying to install the python gdb extensions so I can get the Python stacktrace | |
| 13:05:23 | aspiers | sean-k-mooney: I guess that might work too | |
| 13:07:08 | sean-k-mooney | i think its dieing pretty early in the nova-condoctor console script that is generated via pbr/setup tools | |
| 13:07:41 | aspiers | sean-k-mooney: what's the best way to start it? | |
| 13:08:04 | aspiers | oh wow | |
| 13:08:06 | aspiers | ● devstack@n-sch.service loaded failed failed Devstack devstack@n-sch.service | |
| 13:08:08 | aspiers | ● devstack@n-super-cond.service loaded failed failed Devstack devstack@n-super-cond.service | |
| 13:08:10 | aspiers | ● devstack@s-proxy.service loaded failed failed Devstack devstack@s-proxy.service | |
| 13:08:23 | aspiers | I've had 6 crashes in the last hour | |
| 13:08:27 | sean-k-mooney | i think just "pdb /usr/bin/python /usr/bin/nova-conductor --config-file /etc/nova/nova.conf" maybe with sudo | |
| 13:11:01 | sean-k-mooney | ya one of our deps really dislikes python2.7 apparently | |
| 13:12:22 | sean-k-mooney | was this a clean sles 14 deployment | |
| 13:12:41 | sean-k-mooney | im wondering if i can quickly spin up an container or something and try and repoduce | |
| 13:17:59 | aspiers | gah | |
| 13:18:04 | aspiers | I just lost connection to the node | |
| 13:18:21 | sean-k-mooney | did you check dmesg before | |
| 13:18:33 | sean-k-mooney | there may have been other core dumps or crashes | |
| 13:19:30 | aspiers | coredumpctl shows them all | |
| 13:19:36 | aspiers | all 6 | |
| 13:19:52 | sean-k-mooney | all openstack services | |
| 13:25:08 | aspiers | no, just a few | |
| 13:26:49 | sean-k-mooney | i ment were the 6 core dumps form opensack not did all the openstack services core dump | |
| 13:26:59 | aspiers | yes all openstack I think | |
| 13:27:08 | aspiers | but I can't check because my connection died | |
| 13:27:31 | sean-k-mooney | maybe ssh coredumped... | |
| 13:27:45 | aspiers | nope | |
| 13:27:49 | aspiers | no longer pingable | |
| 13:28:04 | sean-k-mooney | it could be a full kernel panic | |
| 13:28:14 | aspiers | well aren't you the optimist :) | |
| 13:28:24 | sean-k-mooney | do you have acess to the system to check? | |
| 13:28:37 | aspiers | maybe a PSU caught fire, and now the whole DC is a raging inferno | |
| 13:28:59 | aspiers | my colleague is checking via remote console | |
| 13:29:14 | aspiers | I'm going back to figure out why cloud-init is so broken | |
| 13:29:58 | sean-k-mooney | sure :) let me know if they find anything via the remote console | |
| 13:30:13 | aspiers | will do | |
| 13:39:15 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add functional recreate test for bug 1764556 https://review.opendev.org/673812 | |
| 13:39:15 | openstack | bug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,Confirmed] https://launchpad.net/bugs/1764556 | |
| 13:40:28 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add functional regression test for bug 1778305 https://review.opendev.org/673813 | |
| 13:40:28 | openstack | bug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,Confirmed] https://launchpad.net/bugs/1778305 | |
| 13:44:47 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Don't generate service UUID for deleted services https://review.opendev.org/673814 | |
| 13:48:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Add functional recreate test for bug 1764556 https://review.opendev.org/673816 | |
| 13:48:43 | openstack | bug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem) | |
| 13:52:05 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Add functional regression test for bug 1778305 https://review.opendev.org/673818 | |
| 13:52:05 | openstack | bug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem) | |
| 13:54:56 | mriedem | lyarwood: could you poke https://review.opendev.org/#/c/673532/1 and the one above? | |
| 13:57:23 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't generate service UUID for deleted services https://review.opendev.org/673821 | |
| 14:00:39 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for bug 1764556 https://review.opendev.org/673824 | |
| 14:00:39 | openstack | bug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem) | |
| 14:01:23 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional regression test for bug 1778305 https://review.opendev.org/673825 | |
| 14:01:23 | openstack | bug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem) | |
| 14:02:12 | lyarwood | mriedem: yup will try to get to them today | |
| 14:03:37 | mriedem | thanks | |
| 14:04:14 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Don't generate service UUID for deleted services https://review.opendev.org/673827 | |
| 14:20:01 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional recreate test for bug 1764556 https://review.opendev.org/673830 | |
| 14:20:01 | openstack | bug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem) | |
| 14:20:17 | boxiang | mriedem: efried: gibi_off: two patches about live-migration and evacuation need to review https://review.opendev.org/#/c/649963/ https://review.opendev.org/#/c/651969/ | |
| 14:21:20 | mriedem | it'd be nice if mdbooth could look at the evacuate one since he also wrote https://review.openstack.org/#/c/605436/ - would also be nice if cfriesen were around to review those since i know starlingx has patches for these same issues | |
| 14:21:55 | mriedem | i'd prefer a functional test on https://review.opendev.org/#/c/651969/13 since i don't really trust unit tests for this kind of stuff | |
| 14:22:41 | mriedem | gibi is out until next week i think | |
| 14:23:08 | boxiang | ok, so I will add a functional test for this https://review.opendev.org/#/c/651969/13 | |
| 14:23:17 | mdbooth | Meh | |
| 14:23:41 | boxiang | hi mdbooth | |
| 14:23:52 | mdbooth | Duplicated effort is the best effort | |
| 14:24:08 | boxiang | can you take a look for my two patches, thanks | |
| 14:25:44 | mdbooth | boxiang: Not any time soon, unfortunately. Working on stuff that might get used instead. | |
| 14:31:08 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional regression test for bug 1778305 https://review.opendev.org/673832 | |
| 14:31:08 | openstack | bug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem) | |
| 14:31:38 | mriedem | mdbooth: i meant the live migration patch, not the evacuate one | |
| 14:33:52 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Don't generate service UUID for deleted services https://review.opendev.org/673833 | |
| 14:33:56 | sean-k-mooney | by the way https://review.opendev.org/#/c/669738/ has been sitting for a while is mainly functional tests | |
| 14:34:15 | sean-k-mooney | mdbooth: are you ok with this^ | |
| 14:34:25 | sean-k-mooney | mdbooth: i have your patch rebased on top | |
| 14:35:10 | sean-k-mooney | mriedem: i also added functional test as you requested since you last looked | |
| 14:35:19 | mdbooth | sean-k-mooney: Yep. IIRC that's what we discussed previously. | |
| 14:35:42 | sean-k-mooney | i think the backport is still on hold for the orginal change | |
| 14:36:03 | mriedem | dustinc: in case you haven't seen since i didn't vote, i've got some concerns in the base ironic+sdk patch https://review.opendev.org/#/c/642899/33 | |
| 14:36:15 | sean-k-mooney | i belive my change can be applied to stable/X without breaking anything then the other chagne can be stacked on top | |
| 14:36:16 | mriedem | ugly tracebacks in the logs when nova-compute starts up before ironic-api is available | |
| 14:37:59 | dustinc | mriedem: been a bit distracted but did notice and will be looking into it when I get some time. Much appreciated! | |
| 14:58:40 | prometheanfire | I suspect that the new lxml has some ordering issues or something | |
| 14:58:42 | prometheanfire | http://logs.openstack.org/60/673160/3/check/cross-nova-py36/547e78c/testr_results.html.gz | |
| 15:04:49 | openstackgerrit | Merged openstack/nova master: Fix AttributeError in RT._update_usage_from_migration https://review.opendev.org/667687 | |
| 15:04:59 | openstackgerrit | Merged openstack/nova master: Fix RT init arg order in test_unsupported_move_type https://review.opendev.org/667688 | |
| 15:15:31 | openstackgerrit | ZhengMa proposed openstack/nova master: Check container_format on instance creation https://review.opendev.org/673407 | |
| 15:19:12 | sean-k-mooney | prometheanfire: those xml fragment look identical so ya maybe | |
| 15:25:25 | prometheanfire | kk, testing https://review.opendev.org/673848 | |
| 15:33:59 | mriedem | sean-k-mooney: comments in https://review.opendev.org/#/c/669738/ | |
| 15:34:05 | mriedem | maybe i'm just dumb | |
| 15:36:19 | sean-k-mooney | mriedem: thanks ill chack | |
| 15:37:23 | aspiers | sean-k-mooney: the new filter editor in lnav 0.8.5 is awesome (press tab) | |
| 15:45:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Fix AttributeError in RT._update_usage_from_migration https://review.opendev.org/673856 | |
| 15:57:22 | nafiux | Good morning team, I’m trying to configure allocation ratio for CPU and Memory, but I’m getting this error from nova-scheduler: “Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.” | |
| 15:57:30 | openstackgerrit | Merged openstack/nova master: Introduces the openstacksdk to nova https://review.opendev.org/643664 | |