| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-24 | |||
| 17:43:10 | mriedem | ok ok, well, i'm sure mel's boot will find you | |
| 17:44:16 | artom | Mel is clearly a figment of our imagination, because of my previous assertion | |
| 17:46:16 | rabel | sean-k-mooney: it seems that "Intel PCI CI"s pci-test is not non-voting in https://review.openstack.org/#/c/494169/4 . is that correct? can you tell me how to rerun it? | |
| 17:46:33 | mriedem | rabel: it is non-voting | |
| 17:46:41 | mriedem | otherwise there would be a -1 next to 'Verified' | |
| 17:46:51 | mriedem | don't let the red FAILURE on the actual job status trick you | |
| 17:47:07 | rabel | mriedem: ok, thanks. i was confused by having (non-voting) on other tests, but not this one | |
| 17:47:32 | rabel | so no -1 means i'm fine? :) | |
| 17:47:34 | mriedem | ah i don't know how other 3rd party CIs do that | |
| 17:47:44 | mriedem | well, jenkins is the vote you care about | |
| 17:47:45 | mriedem | for CI | |
| 17:47:50 | mriedem | and the vmware CI | |
| 17:48:09 | mriedem | but the latter is on the fritz | |
| 17:48:24 | mriedem | cdent has to call down to larry in the basement to kick it | |
| 17:48:37 | rabel | :D | |
| 17:48:44 | openstackgerrit | Merged openstack/nova master: api-ref: fix key_name note formatting https://review.openstack.org/496718 | |
| 17:48:53 | artom | To be fair, if your patch actually touched on anything PCI, it might be a good idea to see why the Intel CI failed | |
| 17:48:59 | artom | But since it doesn't, you can ignore it | |
| 17:49:17 | openstackgerrit | Merged openstack/nova master: Remove VMware driver _get_vm_ref_from_uuid method https://review.openstack.org/444959 | |
| 17:49:18 | rabel | will it help to write vmware-recheck-patch from time to time if it does not run? | |
| 17:54:39 | openstackgerrit | Evgeny Antyshev proposed openstack/nova master: No scsi unit information in instance XML https://review.openstack.org/495756 | |
| 17:58:35 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: config drive https://review.openstack.org/409404 | |
| 18:04:11 | openstackgerrit | Merged openstack/nova master: Move common definition into common layer https://review.openstack.org/489491 | |
| 18:04:47 | openstackgerrit | Merged openstack/nova master: Remove RamFilter and DiskFilter in default filter https://review.openstack.org/492765 | |
| 18:05:28 | openstackgerrit | Octave Orgeron proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643 | |
| 18:07:00 | mriedem | dansmith: per your comment in alex's change for another test, i think that's covered in https://review.openstack.org/#/c/470578/ | |
| 18:07:25 | dansmith | oh okay I hadn't seen that yeah | |
| 18:07:55 | mriedem | starting the compute runs update_available_resource which does the thing to remove the now deleted allocation | |
| 18:07:57 | mriedem | so yay | |
| 18:23:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for rescheduling during a migration https://review.openstack.org/497541 | |
| 18:24:01 | mriedem | dansmith et al ^ is the functional recreate test for the reschedule + migrate bug | |
| 18:31:28 | mriedem | ugh, you know what, don't we merge allocations for resize to same host in the scheduler | |
| 18:32:15 | mriedem | so i can't remove the allocations for the dest node blindly in prep_resize b/c dest node might == source node for resize to same host, but we also shouldn't leave the allocations as the doubled up ones | |
| 18:33:19 | dansmith | we don't? we have a test for that | |
| 18:33:58 | mriedem | this is where prep_resize fails on the dest host, before we ever confirm/revert | |
| 18:34:07 | mriedem | i think our existing resize to same host tests are all driven on confirm/revert | |
| 18:34:25 | dansmith | right but the scheduler is the thing that doubles | |
| 18:34:31 | mriedem | i know | |
| 18:34:32 | dansmith | that move_claim thingy | |
| 18:34:42 | mriedem | and the compute is what's cleaning up | |
| 18:35:03 | mriedem | in the before times, if this happened, the RT periodic would just overwrite the allocations for the instance based on what it's currently consuming | |
| 18:35:10 | mriedem | are we still doing that? | |
| 18:35:50 | dansmith | what is the thing you think we're not doing? handling the case where prep_resize fails? or the initial doubling before we get that far? | |
| 18:36:21 | mriedem | yeah so before we did the has_ocata_computes thing, the periodic would overwrite the doubled allocation using _update_usage_from_instance | |
| 18:36:26 | mriedem | self.reportclient.update_instance_allocation | |
| 18:36:33 | dansmith | right | |
| 18:36:36 | mriedem | if you don't have ocata computes, that won't correct this now | |
| 18:36:38 | dansmith | which won't happen in pike land | |
| 18:36:49 | mriedem | we're not handling the case that prep_resize fails | |
| 18:36:57 | mriedem | and cleaning up the allocation | |
| 18:36:59 | mriedem | that the scheduler created | |
| 18:37:01 | dansmith | yeah, so you're talking about the case where we've doubled things in the scheduler and don't undouble them if we fail in prep right? | |
| 18:37:07 | mriedem | yup | |
| 18:37:38 | mriedem | so maybe this falls under the same bug i reported for when live migration fails and we don't cleanup | |
| 18:37:56 | dansmith | I would like to point out that if we were doing the allocation thing in the conductor instead of the scheduler, we'd have this all in an auto-cleanup context manager that would roll back the doubling if we failed to kick off a thing | |
| 18:37:58 | mriedem | this is essentially the same kind of fix probably, a periodic checking for failed migrations and cleaning up allocations related to them | |
| 18:38:20 | dansmith | well, we should clean up allocations any time we have a solid failure and know where the instance remains, | |
| 18:38:30 | dansmith | and a failure in prep is that case, right? we know we didn't move anything | |
| 18:39:06 | mriedem | prep_resize is a cast from conductor so i'm not sure how that would auto-cleanup in this case | |
| 18:39:24 | dansmith | it's a cast from conductor to compute? | |
| 18:39:34 | mriedem | yeah | |
| 18:39:41 | dansmith | ah, yeah, I see | |
| 18:39:57 | mriedem | so, remove the dest node allocation when not resizing to same host is simple | |
| 18:40:00 | dansmith | that's legacy from when the api was doing it I think, but.. | |
| 18:40:22 | mriedem | the resize to same host cleanup is shittier, since we basically need to overwrite the allocation back to the original flavor | |
| 18:40:37 | mriedem | which is basically just doing our RT overwrite stuff again, but in a different place | |
| 18:40:38 | dansmith | well, | |
| 18:40:51 | dansmith | we can just subtract what the new flavor would have had in it right? | |
| 18:41:07 | dansmith | not just regenerate, but subtract the new_flavor from our allocation if it's same host | |
| 18:41:17 | dansmith | merge_resources() with new_flavor and -1 as the sign | |
| 18:41:25 | dansmith | that will avoid trampling on shared things | |
| 18:41:30 | mriedem | sure | |
| 18:42:00 | mriedem | god i should probably have a test for the resize to same host case then too... | |
| 18:42:08 | mriedem | and it's nearly 2pm here | |
| 18:42:57 | mriedem | anyway, will work on the easy one first | |
| 18:44:47 | dansmith | mriedem: so you mean the fix for the thing you're testing in 497541 yes? | |
| 18:45:06 | mriedem | yeah | |
| 18:45:24 | mriedem | i don't know if we should do the resize to same host fix in here too or leave that for another bug | |
| 18:45:35 | mriedem | since it'd be a different test | |
| 18:46:00 | dansmith | if it doesn't really overlap then separate is probably best | |
| 18:46:03 | mriedem | and arguably resize to same host failures are less severe for holding up the release | |
| 18:46:12 | dansmith | yes, for real-world people | |
| 18:46:30 | mriedem | the ones on mtv? | |
| 18:46:31 | dansmith | realistically, the way this is going, we're going to be fixing these kinds of issues for a year | |
| 18:46:34 | dansmith | heh | |
| 18:46:40 | mriedem | no shit, this is whack a mole at it's finest | |
| 18:46:59 | dansmith | like, I'm fairly worried about how this is all going to go down | |
| 18:47:17 | cburgess | Worried about what specifically? | |
| 18:47:34 | mriedem | worried about the number of bugs that have shaken out in the last 3 weeks | |
| 18:47:35 | dansmith | cburgess: the thousand places we haven't already found and fixed | |
| 18:47:48 | cburgess | Oh.. so.. normal release cycle? :) | |
| 18:47:57 | mriedem | well, | |
| 18:48:06 | dansmith | cburgess: squared. | |
| 18:48:08 | cburgess | But seriously.. is there something that makes you more concerned this cylce then usual? | |
| 18:48:15 | cburgess | Just the volume of bugs? | |
| 18:48:23 | mriedem | ocata was using placement but not for claims | |
| 18:48:23 | dansmith | cburgess: no specifically what we were just talking about | |
| 18:48:29 | dansmith | cburgess: placement claims stuff | |
| 18:48:38 | cburgess | Ahh ok | |
| 18:48:54 | mriedem | now we're basically redoing the resource tracker with placement, but triplicating it everywhere when things fail | |