| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-20 | |||
| 15:24:06 | zzzeek | jaypipes-ooo: it's an argument im slwoly losing due to the occasional angry customer | |
| 15:24:11 | zzzeek | with a mysql DBA | |
| 15:24:33 | zzzeek | jaypipes-ooo: which of course only matters to me if I have to be on the phone w/ them on friday night b.c. they lost their DB | |
| 15:24:37 | jaypipes-ooo | zzzeek: really no need considering nova, ironic, etc's database schemas are so poor anyway (lots of "blobject" storage and bloated not-really-relational-relations schema that those settings won't IMHO make much of a diff. | |
| 15:24:45 | zzzeek | jaypipes-ooo: agree | |
| 15:39:58 | openstackgerrit | Merged openstack/nova master: libvirt: Use 'virt' as the default machine type for ARMv7 https://review.openstack.org/602592 | |
| 15:40:08 | openstackgerrit | Merged openstack/nova master: Fix evacuate logging https://review.openstack.org/593055 | |
| 15:40:14 | openstackgerrit | Merged openstack/nova stable/queens: Fix soft deleting vm fails after "nova resize" vm https://review.openstack.org/603340 | |
| 15:50:54 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604134 | |
| 15:51:58 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 15:51:59 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 15:56:56 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604138 | |
| 15:58:48 | mnaser | woo | |
| 15:58:54 | mnaser | i got a script going and it helped figure out the issue too | |
| 15:59:49 | mnaser | it's really not cleaned up but does the job, here's a placement 'audit' script: http://paste.openstack.org/show/730450/ | |
| 16:01:15 | mriedem | cool, throw that into "nova-manage placement audit" or something | |
| 16:02:07 | mriedem | should be able to use nova.scheduler.client.report.SchedulerReportClient for the HTTP calls | |
| 16:02:18 | mnaser | yeah, and probably can use nova objects to avoid http calls? | |
| 16:02:38 | mriedem | yeah maybe | |
| 16:03:20 | mnaser | oh boi | |
| 16:03:57 | mnaser | i think i'd say its a bug if a field is missing in nova api if it's empty/blank? | |
| 16:04:12 | mriedem | depends | |
| 16:04:13 | mriedem | which field | |
| 16:04:24 | mnaser | with_servers=True | |
| 16:04:31 | mnaser | the servers: [] field is omitted if no servers are there | |
| 16:04:40 | mnaser | so rather than servers: []. its just not there | |
| 16:04:46 | mriedem | welcome to the compute api | |
| 16:04:53 | mnaser | i.e.: {u'status': u'disabled', u'state': u'down', u'id': u'13b1a4b6-d242-44c5-828e-e6f0a94284da', u'hypervisor_hostname': u'foobar'} | |
| 16:05:51 | mriedem | i believe that's how https://developer.openstack.org/api-ref/compute/#list-hypervisor-servers worked | |
| 16:05:58 | mnaser | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L90-L92 | |
| 16:06:05 | mnaser | if servers with servers = [] evaluates to false? | |
| 16:06:11 | mriedem | yes | |
| 16:06:50 | mnaser | i mean that could have been omitted but i dunno if thats an api bug or something we're expecting | |
| 16:07:14 | mriedem | that's just how that's always worked, | |
| 16:07:21 | mriedem | and this is probably not the only instance of it | |
| 16:07:30 | mriedem | where a field is omitted in the response if it doesn't have any value | |
| 16:07:52 | mriedem | in this case it was just an oversight when i rewrote os-hypervisors in 2.53 | |
| 16:07:59 | mriedem | b/c it's using the same view builder code | |
| 16:08:07 | mnaser | its nice that it gave compute node uuids | |
| 16:08:08 | mnaser | it made this a lot easier | |
| 16:08:29 | mriedem | yup, that's part of why we had to do 2.53 | |
| 16:08:53 | mriedem | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-pike | |
| 16:08:59 | mriedem | it wasn't a fun microversion to write let me tell you | |
| 16:09:24 | mriedem | gmann does have a spec started for fixing random warts like this https://review.openstack.org/#/c/603969/ | |
| 16:10:01 | mriedem | i'll throw this into the etherpad | |
| 16:11:54 | openstackgerrit | caoyuan proposed openstack/nova master: Option "scheduler_default_filters" is deprecated. https://review.openstack.org/604148 | |
| 16:12:23 | mnaser | holy crap | |
| 16:12:39 | mnaser | i just ran this on a much bigger deployment and its a terrifying amount of stuff that is misreported | |
| 16:13:10 | mnaser | deleted vms, rps missing, and incorrect assignments | |
| 16:13:58 | mriedem | so you have orphaned allocations and rps for deleted things in nova yes? | |
| 16:14:06 | mriedem | likely because of that fix referenced earlier | |
| 16:14:15 | mriedem | incorrect assignments is more troubling | |
| 16:14:30 | tobias-urdin | i've had the same after bombing clouds with testing | |
| 16:14:58 | mnaser | mriedem: im assuming its live migrations not cleaning up | |
| 16:15:08 | mriedem | failed live migrations? | |
| 16:15:22 | mnaser | i dont really know, this cloud is been around for a while so i wouldnt know | |
| 16:15:39 | mnaser | pike though | |
| 16:15:41 | mnaser | queens | |
| 16:15:41 | mnaser | err no | |
| 16:15:42 | mriedem | we definitely have functional tests for successful and failed live migrations where we assert the allocations are in the right place once we're done | |
| 16:16:06 | mnaser | 108 deleted vms that are in placement | |
| 16:16:08 | mriedem | at pike GA i think there were quite a few holes like that though for move operations | |
| 16:16:28 | mnaser | 8 missing RPs (but that is probably related to the bug we just spoke about and doesnt hurt that much) | |
| 16:16:43 | mnaser | 6 'incorrect' assignments | |
| 16:16:54 | mriedem | of those 108 deleted vms still in placement, | |
| 16:17:02 | mriedem | i'd be willing to bet a lot of those were local deletes in the API? | |
| 16:17:08 | mriedem | which was also fixed later | |
| 16:17:22 | mnaser | yeah, i mean i can do some checking if you want me to do that before deeleting | |
| 16:18:08 | mriedem | https://review.openstack.org/#/q/If507e23f0b7e5fa417041c3870d77786498f741d | |
| 16:18:23 | mriedem | https://review.openstack.org/#/c/580498/ isn't even released yet on pike | |
| 16:18:47 | dansmith | sheesh | |
| 16:19:06 | mriedem | merged july 9... | |
| 16:19:42 | mriedem | mnaser: so that 108 and 8 orphaned things are pretty safe to cleanup i think | |
| 16:20:08 | mriedem | the 6 incorrect will likely require a bit of investigation, but could just see where is the instance now and compare to what it's allocations are | |
| 16:24:30 | jroll | jaypipes-ooo: way late, but I just found out we also have a john snow in our org (cc stephenfin) | |
| 16:37:33 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Fix upper-constraints link in tox file https://review.openstack.org/604158 | |
| 16:39:07 | larsks | mnaser: that has been useful twice so far :) | |
| 16:39:21 | mnaser | larsks: meanwhile i just spent most of the past few hours doing the same thing | |
| 16:39:36 | mnaser | but in a different approach that does list all servers + all tenants because that would burn down some oof our deployments | |
| 16:47:58 | mriedem | dansmith: you know how we added a microversion to make live migration an async rpc cast from api to conductor? | |
| 16:48:07 | mriedem | i'm kind of wondering if we should do something similar for resize | |
| 16:48:13 | mnaser | mriedem: hacked some more, http://paste.openstack.org/show/730450/ generates http://paste.openstack.org/show/730460/ | |
| 16:48:25 | mnaser | ill test it out a bit more and try to see if i can add it to nova-manage | |
| 16:48:31 | mriedem | with resize we only finally return the api response once conductor casts to prep_resize on compute | |
| 16:48:35 | dansmith | mriedem: is resize not async already? | |
| 16:48:39 | mriedem | nope | |
| 16:48:40 | dansmith | because it could take a long ass time | |
| 16:48:40 | openstackgerrit | Merged openstack/nova master: Merge extended_volumes extension response into server view builder https://review.openstack.org/596285 | |
| 16:48:43 | mriedem | well, | |
| 16:48:47 | openstackgerrit | Merged openstack/nova stable/rocky: Update RequestSpec.flavor on resize_revert https://review.openstack.org/602403 | |
| 16:48:49 | mriedem | it's async once we pick a compute and cast to it | |
| 16:48:57 | dansmith | ah, okay | |
| 16:49:12 | mriedem | the live migration task before that async microversion would do the reschedules and such within conductor | |
| 16:49:16 | mriedem | which could timeout the api response | |
| 16:49:57 | mriedem | for cross-cell resize, i'm considering doing similar to the pre-migration checks *from* conductor and going back to the scheduler well for more hosts if the one selected doesn't pass the pre-check | |
| 16:50:06 | mriedem | *pre-live migration checks | |
| 16:50:34 | mriedem | likely a todo to throw on the pile for my patch at this point | |
| 16:50:39 | dansmith | ack | |
| 16:50:51 | mriedem | i have officially started losing sleep over this | |
| 16:51:16 | mriedem | need to contract out some hot alaski action | |