| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-20 | |||
| 17:09:54 | mnaser | mriedem: working around it by telling folks to not set baremetal machines into maintenance mode | |
| 17:13:37 | sean-k-mooney | mnaser: does that not require you to use the iroic api and therefor have admin acess | |
| 17:14:05 | sean-k-mooney | so what everyou your working aound i assume you can jsut tell the vxhost admins dont do that untill its fixed | |
| 17:14:46 | mnaser | sean-k-mooney: yes exactly its more of an operator side workaround so its a lot easier than a user facing workaround | |
| 17:15:02 | sean-k-mooney | mnaser: what the issue out of interest | |
| 17:15:41 | mnaser | sean-k-mooney: setting a node to 'in maintenance' makes it not show up anymore in the api request to get active nodes from nova-compute so nova marks tyhe service as deleted | |
| 17:16:23 | mnaser | but if you remove the flag, the node comes back up, and then it fails to be added to the db because there is already a record with the same uuid | |
| 17:16:27 | sean-k-mooney | oh ok and are you getting duplicate errors when its marked as no longer in maintas in placemnet | |
| 17:16:45 | sean-k-mooney | ya ok that makes sense | |
| 17:16:51 | mnaser | yeah, this is because of a recent change where uuids now uses ironic uuid | |
| 17:16:54 | sean-k-mooney | it also feel like that is pretty latent behavior | |
| 17:17:12 | sean-k-mooney | recent as in rocky ish | |
| 17:17:19 | mnaser | yes that was a recent addition | |
| 17:17:31 | mnaser | https://review.opendev.org/#/c/571535/ rocky indeed | |
| 17:17:40 | sean-k-mooney | ya i dont think we have many customer deployint that yet | |
| 17:17:44 | sean-k-mooney | most are on queens | |
| 17:18:17 | sean-k-mooney | well i lie most are proably on newton untill that is EOL in deceber then most will be on queens | |
| 18:06:10 | openstackgerrit | Merged openstack/nova master: Add functional recreate test for bug 1823370 https://review.opendev.org/650424 | |
| 18:06:11 | openstack | bug 1823370 in OpenStack Compute (nova) "Evacuations are not restricted to the source cell during scheduling" [Medium,In progress] https://launchpad.net/bugs/1823370 - Assigned to Matt Riedemann (mriedem) | |
| 19:06:52 | donnyd | mnaser: just ran into this issue this morning on FN | |
| 19:08:24 | donnyd | A node didn't want to clean, was thrown into maintenance and then had to pull it from nova db to get it schedulable again | |
| 19:11:25 | mnaser | yep | |
| 19:11:51 | openstackgerrit | Merged openstack/python-novaclient master: Microversion 2.75 - Multiple API cleanup changes https://review.opendev.org/676275 | |
| 19:28:53 | openstackgerrit | Merged openstack/python-novaclient master: API microversion 2.76: Add 'power-update' external event https://review.opendev.org/666792 | |
| 19:58:36 | mriedem | efried: i guess t-2 should be closed out in lp huh | |
| 19:58:39 | mriedem | as of july 25 | |
| 20:00:16 | mriedem | i closed out the bps for 2.75 and 2.76 | |
| 20:07:49 | efried | mriedem: I don't know what you're saying. | |
| 20:08:04 | mriedem | efried: the t-2 milestone in launchpad for nova is still open | |
| 20:08:13 | mriedem | even though we're past the t-2 (fake-ish) milestone | |
| 20:08:24 | mriedem | https://wiki.openstack.org/wiki/Nova/Train_Release_Schedule | |
| 20:08:36 | mriedem | melwitt: https://review.opendev.org/#/c/672840/8 is busted - you remove the call to archive | |
| 20:08:38 | mriedem | *removed | |
| 20:08:56 | mriedem | i think my comment in PS4 caused the confusion, i just meant to remove the $conf passed to the bash function since we didn't need it anymore | |
| 20:09:37 | efried | mriedem: I didn't know launchpad tracked "milestones". How do I look at that? | |
| 20:09:57 | efried | (and, not to put too fine a point on it, but does anyone care?) | |
| 20:10:14 | mriedem | https://launchpad.net/nova/+series | |
| 20:10:55 | mriedem | efried: not really since the release team stopped doing milestone releases, | |
| 20:11:01 | mriedem | we really only have them for internal tracking | |
| 20:11:05 | efried | so... I can go into it and uncheck Active and that's it? | |
| 20:11:11 | mriedem | i.e. t-2, nova-spec freeze | |
| 20:11:41 | mriedem | in the past i've created a release | |
| 20:11:46 | mriedem | which closes it out | |
| 20:11:54 | melwitt | mriedem: argh, sorry. I saw the commands were in a function and then I promptly didn't realize that was the func call. will fix | |
| 20:12:15 | mriedem | efried: i'm not sure if marking it as inactive just removes it from targeting for bugs and blueprints but still shows up in history | |
| 20:12:28 | mriedem | since we don't create an actual release (we don't tag the repo), marking as not active is probably more correct | |
| 20:13:06 | efried | made it disappear from the little flow charty thingy. | |
| 20:17:35 | mriedem | i still see it here https://launchpad.net/nova/train | |
| 20:17:42 | mriedem | but not marked active https://launchpad.net/nova/+milestone/train-2 | |
| 20:17:44 | mriedem | which means, | |
| 20:18:05 | mriedem | it doesn't show up as a milestone target either, which is what we want | |
| 20:18:09 | mriedem | so we're good | |
| 20:23:03 | efried | \o/ | |
| 20:38:26 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Remove unnecessary try-catch around 'getCPUMap' https://review.opendev.org/671790 | |
| 20:38:26 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: Rename 'nova.objects.instance_numa_topology' https://review.opendev.org/671789 | |
| 20:38:27 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Remove unnecessary argument https://review.opendev.org/674396 | |
| 20:38:28 | openstackgerrit | Eric Fried proposed openstack/nova master: claims: Remove useless caching https://review.opendev.org/671791 | |
| 20:38:28 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.opendev.org/537414 | |
| 20:38:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | |
| 20:38:29 | openstackgerrit | Eric Fried proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | |
| 20:38:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | |
| 20:38:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | |
| 20:38:31 | openstackgerrit | Eric Fried proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | |
| 20:38:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565 | |
| 20:38:32 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 20:38:32 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 20:38:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Add '[compute] cpu_dedicated_set' option https://review.opendev.org/671792 | |
| 20:38:33 | openstackgerrit | Eric Fried proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | |
| 20:38:34 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 20:38:34 | openstackgerrit | Eric Fried proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 20:38:35 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | |
| 20:38:46 | melwitt | mriedem: the archive command should go after 'set -e' too right? or no? | |
| 20:39:29 | melwitt | oh, it shouldn't bc we expect it to return 1 several times | |
| 20:40:54 | mriedem | melwitt: i think it can because the functions will only return a non-0 if they fail | |
| 20:41:19 | melwitt | mriedem: oh, I see. the function, not the individual commands. ok thanks | |
| 20:41:21 | mriedem | that's one issue i've meant to cleanup in that script is if the archive or purge fails the script wouldn't actually fail and the job wouldnt' fail | |
| 20:42:29 | melwitt | yeah, I see | |
| 20:42:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 20:45:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 20:45:48 | mriedem | sean-k-mooney: stephenfin: i've addressed comments in ^ and am +2 now | |
| 20:48:34 | openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | |
| 20:55:39 | mriedem | efried: looking at the grenade failures in https://review.opendev.org/#/c/676684/, it appears to be segfaults killing q-svc | |
| 20:55:59 | mriedem | Aug 20 15:53:30 ubuntu-bionic-rax-dfw-0010301045 kernel: /usr/bin/python[21875]: segfault at a9 ip 000055aee57ec9e4 sp 00007ffed66329d8 error 4 in python2.7[55aee5716000+301000] | |
| 20:56:05 | mriedem | which i've seen locally but not so much in the gate | |
| 20:56:11 | efried | mriedem: segfault + switch-to-something-called-binary == suspicious | |
| 20:56:34 | mriedem | but these jobs aren't even using postgresql | |
| 20:56:50 | efried | I didn't go back and check the previous failures | |
| 20:57:02 | efried | but it seemed like *all* the grenade jobs failed *every* patch set. | |
| 20:57:08 | efried | (anecdotal ^ ) | |
| 20:58:07 | mriedem | the pips are installed | |
| 20:58:08 | efried | Anyway, I have a definite -1 on the requirements-check problem. If the grenade thing turns out to be unrelated, great. | |
| 20:58:08 | mriedem | psycopg2==2.8.3 psycopg2-binary==2.8.3 | |
| 20:58:26 | mriedem | the good news is there are hits in logstash | |
| 20:58:29 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unexpected%20exception%20in%20API%20method%3A%20ConnectFailure%3A%20Unable%20to%20establish%20connection%20to%5C%22%20AND%20message%3A%5C%22v2.0%2Fports%3Fdevice_id%5C%22%20AND%20message%3A%5C%22('Connection%20aborted.'%2C%20BadStatusLine('No%20status%20line%20received%20-%20the%20server%20has%20closed%20the%20connection'%2C))%5C%22%20AND%20tags | |
| 20:58:30 | mriedem | 5C%22screen-n-api.txt%5C%22&from=7d | |
| 20:58:35 | mriedem | the bad news is they are only on that one change | |
| 20:58:44 | efried | is that bad news? | |
| 20:58:56 | efried | It seems like it would be further evidence that the problem is *there*. | |
| 21:02:45 | mriedem | well it's not good news | |
| 21:03:19 | mriedem | i've seen this same thing in my devstack vms i've created lately b/c there is too much filling up a single 8gb ram/8vpu vm even after stripping the services to pretty bare bone | |