| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-21 | |||
| 16:44:24 | mriedem | 2. update_available_resource runs on dest host to start tracking incoming migration | |
| 16:44:29 | mriedem | kaboom | |
| 16:44:52 | mriedem | 3. claim happens on dest host to create migration_context (on a different request thread) | |
| 16:45:50 | dansmith | mriedem: yeah, makes sense.. if we're looking for an instance involved with a migration, it's likely we care about the migration context | |
| 16:47:04 | mriedem | dansmith: ok will push something | |
| 16:47:38 | efried | mriedem: are you working up a func test? | |
| 16:47:43 | mriedem | nope | |
| 16:48:11 | efried | is a func test possible? | |
| 16:48:41 | mriedem | yeah probably | |
| 16:48:52 | mriedem | i mean, i *could* write one up | |
| 16:49:10 | mriedem | if $$$properly$$$ motivated | |
| 16:52:07 | openstackgerrit | Dan Smith proposed openstack/nova master: Add image precaching docs for aggregates https://review.opendev.org/687348 | |
| 16:52:24 | efried | I just can't seem to stop seeing that ^ as "preaching" | |
| 16:52:29 | dansmith | mriedem: ^ includes some generic image caching doc stuff in addition to precaching | |
| 16:52:56 | dansmith | as we discussed | |
| 16:53:56 | mriedem | is it worth marking "Partial-Bug: #1847302" or at least related? | |
| 16:53:56 | openstack | bug 1847302 in OpenStack Compute (nova) "doc: need admin guide for the image cache" [Undecided,New] https://launchpad.net/bugs/1847302 | |
| 16:54:09 | dansmith | if you want | |
| 16:54:21 | mriedem | i think that woudl be good (Related-Bug) | |
| 16:54:28 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Don't populate resources for not-yet-migrated instance https://review.opendev.org/689842 | |
| 16:54:31 | efried | mriedem: ^ | |
| 16:54:33 | openstackgerrit | Dan Smith proposed openstack/nova master: Add image precaching docs for aggregates https://review.opendev.org/687348 | |
| 16:54:54 | dansmith | mriedem: already did partial-, I can change to related- if you want, as I'm sure this will need some respins | |
| 16:55:10 | mriedem | up to you, only difference is with partial the bug is assigned to you | |
| 16:55:21 | mriedem | which might imply signing up for more work than you want | |
| 16:56:36 | dansmith | drat! :) | |
| 17:01:22 | efried | still trying to understand the race. | |
| 17:01:22 | efried | - create instance | |
| 17:01:22 | efried | - migrate instance, which has the following steps: | |
| 17:01:22 | efried | - x create migration context -- this creates the migration record and populates the instance in it, but the instance doesn't have a migration_context yet | |
| 17:01:22 | efried | - y schedule | |
| 17:01:22 | efried | - z claim on destination (_move_claim) -- this is the thing that sets the instance's migration_context | |
| 17:01:22 | efried | So we need update_available_resource on the dest compute to run after x and before z, yah? | |
| 17:03:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Join migration_context and flavor in Migration.instance https://review.opendev.org/689846 | |
| 17:03:53 | mriedem | your steps are wrong | |
| 17:04:32 | mriedem | you don't control when update_available_resource runs, it's on a timer in a thread pool | |
| 17:04:58 | efried | well, we can trigger it manually in a test | |
| 17:05:34 | mriedem | 1. create instance, | |
| 17:06:02 | mriedem | 2. wrap _prep_resize on the dest compute service to trigger the periodic on the dest when calling _prep_resize | |
| 17:06:08 | mriedem | 3. initiate a migration | |
| 17:06:11 | mriedem | check logs for TypeError | |
| 17:06:54 | dansmith | this is happening in CI yeah? | |
| 17:07:04 | dansmith | so more soak time and we would have found it I guess | |
| 17:07:07 | mriedem | yeah, multinode migration tests | |
| 17:09:42 | mriedem | efried: i can whip up a func recreate test quick if you want | |
| 17:09:43 | efried | can we please merge https://review.opendev.org/#/c/686207/ which will be useful for this? | |
| 17:09:54 | mriedem | no | |
| 17:09:55 | efried | mriedem: I'm working on it, wouldn't mind the experience. | |
| 17:10:01 | mriedem | because we don't want to backport that spy stuff | |
| 17:10:05 | mriedem | and this has to go to train | |
| 17:10:12 | efried | ight | |
| 17:10:34 | mriedem | i mean, i'd rather not backport the spy stuff | |
| 17:10:44 | mriedem | if enough stuff builds on it i guess it will eventually happen | |
| 17:11:38 | efried | slight catch-22 there | |
| 17:12:00 | mriedem | well, it would be nice if the spy stuff does land, that it soaks a bit before we have to backport it for something that otherwise wouldn't need it | |
| 17:12:35 | mriedem | cuz if there is a bug in spy then you have to fix and backport that later as well | |
| 17:12:37 | efried | okay. Its successor has soakage. Let's merge both | |
| 17:13:02 | mriedem | i don't think you can claim "soakage" on anything that isn't merged | |
| 17:13:06 | dansmith | to me, confidence in soakage comes from being in the firehose, not a few rechecks on a single patch | |
| 17:13:10 | dansmith | yeah, that ^ | |
| 17:16:06 | efried | no, my point is, if we merge both of those things in master now, they're in the firehose because the second patch puts them in the way of I think three functional tests. Then we can soak them in master until you're comfortable before doing anything further. | |
| 17:26:22 | openstackgerrit | Merged openstack/nova stable/rocky: Stop sending bad values from libosinfo to libvirt https://review.opendev.org/688068 | |
| 17:26:27 | openstackgerrit | Merged openstack/nova stable/stein: Fix unit of hw_rng:rate_period https://review.opendev.org/689153 | |
| 17:42:35 | artom | sean-k-mooney, btw, do we care about https://review.opendev.org/#/c/465783/ not being in OSP10 Neutron? | |
| 17:42:56 | artom | (Sorry to make you context-switch - it's a dependency of https://code.engineering.redhat.com/gerrit/#/c/183568/1) | |
| 17:43:08 | artom | Doh, was meant for the internal channel | |
| 17:48:15 | efried | mriedem: I'm close-ish. | |
| 17:48:15 | efried | Doing the mock at _prep_resize wasn't working because both the migration and the migration context are happening underneath that. | |
| 17:48:15 | efried | AFAICT I need to get inside _move_claim itself, between _claim_existing_migration (at the top) and the init of mig_context (at the bottom). | |
| 17:48:15 | efried | So I tried mocking _claim_existing_migration to do its thing first, followed by running periodics. | |
| 17:48:15 | efried | But that deadlocks because *in test* they're both running in the same worker, so they're under the same COMPUTE_RESOURCE_SEMAPHORE. | |
| 17:48:15 | efried | Where would you go from here? | |
| 18:11:51 | mriedem | efried: you don't need to replace _prep_resize entirely, just wrap it, | |
| 18:12:08 | mriedem | i.e. stub it out and inside the stub run the periodics and then call the original _prep_resize | |
| 18:12:28 | mriedem | like this https://review.opendev.org/#/c/689013/1/nova/tests/functional/regressions/test_bug_1848343.py | |
| 18:19:34 | mriedem | looks like the ceph job is not happy again | |
| 18:19:34 | mriedem | http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 | |
| 18:19:53 | mriedem | and the graphs are outdated on master since we don't run tempest-full-py3 in master (we run tempest-integrated-compute) | |
| 18:20:36 | melwitt | dangit, I'll look and also get the grafana page issue fixed | |
| 18:20:56 | mriedem | i looked at a ceph failure last week and it looked like some setup issue in devstack | |
| 18:21:48 | melwitt | ok | |
| 18:23:53 | openstackgerrit | sean mooney proposed openstack/nova master: block rebuild when numa topology changed https://review.opendev.org/687957 | |
| 18:23:53 | openstackgerrit | sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861 | |
| 18:26:59 | openstackgerrit | Merged openstack/nova stable/train: Update compute rpc version alias for train https://review.opendev.org/689164 | |
| 18:27:05 | openstackgerrit | Merged openstack/nova stable/train: Error out interrupted builds https://review.opendev.org/687216 | |
| 18:28:23 | melwitt | mriedem: can you link me a failure? I don't see any recent failure when I open a few sample patches | |
| 18:33:48 | dansmith | melwitt: could probably convince kibana to show you just that job with failed status | |
| 18:34:17 | melwitt | oh yeah, kibana. my old friend | |
| 18:35:53 | dansmith | something like build_status:FAILURE build_name:"ceph" | |
| 18:35:57 | openstackgerrit | Merged openstack/nova stable/train: Handle get_host_availability_zone error during reschedule https://review.opendev.org/686226 | |
| 18:36:17 | melwitt | yeah, thanks | |
| 18:36:39 | melwitt | can do that | |
| 18:50:21 | openstackgerrit | Merged openstack/nova stable/train: Fix exception translation when creating volume https://review.opendev.org/688072 | |
| 19:12:57 | efried | mriedem: yes, that's what I did, but it didn't trigger the problem. | |
| 19:14:10 | jmlowe | Is anybody aware off the top of their heads of a mechanism to quota vgpu's ? | |
| 19:17:29 | efried | melwitt: ^ did we get anywhere with the placement-based quota thing? | |
| 19:17:56 | melwitt | efried: not yet. that's johnthetubaguy's unified limits spec | |
| 19:18:24 | melwitt | I had a chat with him about it and he'd like to get started on it this cycle and will re-propose the spec | |
| 19:18:34 | efried | https://review.opendev.org/#/c/602201/ | |
| 19:18:38 | melwitt | (and I will help with the work) | |