| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-31 | |||
| 12:28:37 | aspiers | default is 3.4.6 | |
| 12:28:42 | sean-k-mooney | 3.4 might work | |
| 12:28:47 | sean-k-mooney | we just never offially supported it | |
| 12:28:58 | aspiers | ok | |
| 12:28:58 | sean-k-mooney | we went strait to 3.5 | |
| 12:32:14 | stephenfin | gmann: Could you take a look at https://review.opendev.org/#/c/661391/ (DevStack stuff) when you've a chance? | |
| 12:34:23 | aspiers | sean-k-mooney: https://github.com/tstack/lnav/issues/436#issuecomment-516829106 | |
| 12:36:18 | gmann | stephenfin: sure. do you have nova patch to test that ? only thing i want to verify is IPv6 job i added recently on nova gate works fine. I removed the setting of novnc addresses from base devstack job and let /lib/nova handle that. | |
| 12:36:32 | gmann | stephenfin: of let me add the same job on devstack on top of that patch which will test both. | |
| 12:36:37 | gmann | s/of/or | |
| 12:46:02 | aspiers | sean-k-mooney: I have a coredump now | |
| 12:46:28 | aspiers | sean-k-mooney: also https://github.com/aspiers/lnav-formats/blob/master/oslo_log.json | |
| 12:46:41 | aspiers | although I'm not sure that was necessary, it might be in other repos | |
| 12:51:38 | stephenfin | gmann: Yeah, I think this one should do it https://review.opendev.org/#/c/667388/ | |
| 12:54:03 | gmann | stephenfin: thanks. rechecked that. | |
| 12:56:10 | sean-k-mooney | aspiers: with python 3.4? | |
| 12:56:18 | aspiers | no | |
| 12:56:29 | aspiers | http://paste.openstack.org/show/755163/ | |
| 12:57:14 | sean-k-mooney | aspiers: also did you add oslo_log support to lnav as a plugin if so i might have to try that | |
| 12:57:31 | aspiers | sean-k-mooney: that's what the link above is | |
| 12:57:46 | aspiers | sean-k-mooney: it gets installed via "lnav -i extra" | |
| 12:57:56 | aspiers | just type that | |
| 12:58:02 | sean-k-mooney | ya i clicked and got confused with your core dump comment but imm looking at teh paste now | |
| 13:03:00 | sean-k-mooney | hum well i cant really tell what cased that seg fault other thin it appers to be a result of a getattr call on a randomp python object. so ill blame eventlest by default if you can repoduce with the pytho debuger instad of gdb that might help | |
| 13:05:00 | sean-k-mooney | aspiers: can you try running the conductor with pdb | |
| 13:05:03 | aspiers | sean-k-mooney: I'm trying to install the python gdb extensions so I can get the Python stacktrace | |
| 13:05:23 | aspiers | sean-k-mooney: I guess that might work too | |
| 13:07:08 | sean-k-mooney | i think its dieing pretty early in the nova-condoctor console script that is generated via pbr/setup tools | |
| 13:07:41 | aspiers | sean-k-mooney: what's the best way to start it? | |
| 13:08:04 | aspiers | oh wow | |
| 13:08:06 | aspiers | ● devstack@n-sch.service loaded failed failed Devstack devstack@n-sch.service | |
| 13:08:08 | aspiers | ● devstack@n-super-cond.service loaded failed failed Devstack devstack@n-super-cond.service | |
| 13:08:10 | aspiers | ● devstack@s-proxy.service loaded failed failed Devstack devstack@s-proxy.service | |
| 13:08:23 | aspiers | I've had 6 crashes in the last hour | |
| 13:08:27 | sean-k-mooney | i think just "pdb /usr/bin/python /usr/bin/nova-conductor --config-file /etc/nova/nova.conf" maybe with sudo | |
| 13:11:01 | sean-k-mooney | ya one of our deps really dislikes python2.7 apparently | |
| 13:12:22 | sean-k-mooney | was this a clean sles 14 deployment | |
| 13:12:41 | sean-k-mooney | im wondering if i can quickly spin up an container or something and try and repoduce | |
| 13:17:59 | aspiers | gah | |
| 13:18:04 | aspiers | I just lost connection to the node | |
| 13:18:21 | sean-k-mooney | did you check dmesg before | |
| 13:18:33 | sean-k-mooney | there may have been other core dumps or crashes | |
| 13:19:30 | aspiers | coredumpctl shows them all | |
| 13:19:36 | aspiers | all 6 | |
| 13:19:52 | sean-k-mooney | all openstack services | |
| 13:25:08 | aspiers | no, just a few | |
| 13:26:49 | sean-k-mooney | i ment were the 6 core dumps form opensack not did all the openstack services core dump | |
| 13:26:59 | aspiers | yes all openstack I think | |
| 13:27:08 | aspiers | but I can't check because my connection died | |
| 13:27:31 | sean-k-mooney | maybe ssh coredumped... | |
| 13:27:45 | aspiers | nope | |
| 13:27:49 | aspiers | no longer pingable | |
| 13:28:04 | sean-k-mooney | it could be a full kernel panic | |
| 13:28:14 | aspiers | well aren't you the optimist :) | |
| 13:28:24 | sean-k-mooney | do you have acess to the system to check? | |
| 13:28:37 | aspiers | maybe a PSU caught fire, and now the whole DC is a raging inferno | |
| 13:28:59 | aspiers | my colleague is checking via remote console | |
| 13:29:14 | aspiers | I'm going back to figure out why cloud-init is so broken | |
| 13:29:58 | sean-k-mooney | sure :) let me know if they find anything via the remote console | |
| 13:30:13 | aspiers | will do | |
| 13:39:15 | openstack | bug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,Confirmed] https://launchpad.net/bugs/1764556 | |
| 13:39:15 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add functional recreate test for bug 1764556 https://review.opendev.org/673812 | |
| 13:40:28 | openstack | bug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,Confirmed] https://launchpad.net/bugs/1778305 | |
| 13:40:28 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add functional regression test for bug 1778305 https://review.opendev.org/673813 | |
| 13:44:47 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Don't generate service UUID for deleted services https://review.opendev.org/673814 | |
| 13:48:43 | openstack | bug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem) | |
| 13:48:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Add functional recreate test for bug 1764556 https://review.opendev.org/673816 | |
| 13:52:05 | openstack | bug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem) | |
| 13:52:05 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Add functional regression test for bug 1778305 https://review.opendev.org/673818 | |
| 13:54:56 | mriedem | lyarwood: could you poke https://review.opendev.org/#/c/673532/1 and the one above? | |
| 13:57:23 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't generate service UUID for deleted services https://review.opendev.org/673821 | |
| 14:00:39 | openstack | bug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem) | |
| 14:00:39 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for bug 1764556 https://review.opendev.org/673824 | |
| 14:01:23 | openstack | bug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem) | |
| 14:01:23 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional regression test for bug 1778305 https://review.opendev.org/673825 | |
| 14:02:12 | lyarwood | mriedem: yup will try to get to them today | |
| 14:03:37 | mriedem | thanks | |
| 14:04:14 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Don't generate service UUID for deleted services https://review.opendev.org/673827 | |
| 14:20:01 | openstack | bug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem) | |
| 14:20:01 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional recreate test for bug 1764556 https://review.opendev.org/673830 | |
| 14:20:17 | boxiang | mriedem: efried: gibi_off: two patches about live-migration and evacuation need to review https://review.opendev.org/#/c/649963/ https://review.opendev.org/#/c/651969/ | |
| 14:21:20 | mriedem | it'd be nice if mdbooth could look at the evacuate one since he also wrote https://review.openstack.org/#/c/605436/ - would also be nice if cfriesen were around to review those since i know starlingx has patches for these same issues | |
| 14:21:55 | mriedem | i'd prefer a functional test on https://review.opendev.org/#/c/651969/13 since i don't really trust unit tests for this kind of stuff | |
| 14:22:41 | mriedem | gibi is out until next week i think | |
| 14:23:08 | boxiang | ok, so I will add a functional test for this https://review.opendev.org/#/c/651969/13 | |
| 14:23:17 | mdbooth | Meh | |
| 14:23:41 | boxiang | hi mdbooth | |
| 14:23:52 | mdbooth | Duplicated effort is the best effort | |
| 14:24:08 | boxiang | can you take a look for my two patches, thanks | |
| 14:25:44 | mdbooth | boxiang: Not any time soon, unfortunately. Working on stuff that might get used instead. | |
| 14:31:08 | openstack | bug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem) | |
| 14:31:08 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional regression test for bug 1778305 https://review.opendev.org/673832 | |
| 14:31:38 | mriedem | mdbooth: i meant the live migration patch, not the evacuate one | |
| 14:33:52 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Don't generate service UUID for deleted services https://review.opendev.org/673833 | |
| 14:33:56 | sean-k-mooney | by the way https://review.opendev.org/#/c/669738/ has been sitting for a while is mainly functional tests | |
| 14:34:15 | sean-k-mooney | mdbooth: are you ok with this^ | |
| 14:34:25 | sean-k-mooney | mdbooth: i have your patch rebased on top | |
| 14:35:10 | sean-k-mooney | mriedem: i also added functional test as you requested since you last looked | |
| 14:35:19 | mdbooth | sean-k-mooney: Yep. IIRC that's what we discussed previously. | |
| 14:35:42 | sean-k-mooney | i think the backport is still on hold for the orginal change | |