| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-19 | |||
| 17:12:04 | openstack | Launchpad bug 1814916 in Cinder "MessagingTimeout in cinder-api during volume attachment update due to VolumeAttachmentNotFound in cinder-volume" [Medium,Confirmed] | |
| 17:12:19 | mriedem | yeah i don't know what's up with that volume attachment not found one | |
| 17:12:31 | mriedem | the api finds the record and then c-vol blows up b/c it's gone from the db, so clearly a race somewhere | |
| 17:12:57 | mriedem | probably just need someone to trace requests in a log to see when the PUT request is made and when the DELETE request is made | |
| 17:12:57 | cfriesen | mriedem: smcginnis: but after fixing that I'm hitting the "Guest refused to detach" issue. I'm guessing it's related to stalls we're seeing in the guest, but still digging into causes. | |
| 17:14:26 | openstackgerrit | Adam Spiers proposed openstack/nova master: WIP: Convert driver supported capabilities to compute node provider traits https://review.openstack.org/538498 | |
| 17:15:11 | aspiers | mriedem, efried: I've just finished the first version of the tests which were missing for capabilities to traits: https://review.openstack.org/#/c/538498/12/nova/tests/functional/test_servers.py | |
| 17:15:28 | mriedem | etcd changes for devstack backported https://review.openstack.org/#/q/topic:bug/1810526+(status:open+OR+status:merged) | |
| 17:15:43 | aspiers | mriedem, efried: I'm sure you'll find plenty of things to improve, but hopefully this is a good base | |
| 17:16:21 | aspiers | not sure if I went overkill on the tests for standard vs. custom traits, and capabilities vs. update_provider_tree | |
| 17:22:42 | aspiers | mriedem: BTW I kept you as the author and added myself via Co-Authored-By - was that the right thing to do? | |
| 17:22:51 | aspiers | not sure what the convention is here | |
| 17:24:18 | mriedem | aspiers: yup | |
| 17:24:28 | mriedem | more testing > less testing | |
| 17:24:50 | aspiers | mriedem: also, should I put "related to blueprint expose-host-capabilities" in the commit message? | |
| 17:32:48 | mriedem | hmm, that's not approved https://blueprints.launchpad.net/nova/+spec/expose-host-capabilities but the change is essentially trying to solve that same problem | |
| 17:32:50 | mriedem | so sure | |
| 17:33:26 | mriedem | the spec for that bp predates traits https://review.openstack.org/#/c/222200/ | |
| 17:33:35 | mriedem | so it was trying to solve it with something different | |
| 17:43:32 | sean-k-mooney | adrianc: o/ are you wokring on a new version of https://review.openstack.org/#/c/620115/ | |
| 17:44:05 | sean-k-mooney | adrianc: if not i might start adressing stephenfin nits after i get something to eat. | |
| 17:54:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance to make place for port healing https://review.openstack.org/637953 | |
| 17:54:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance (2) https://review.openstack.org/637954 | |
| 17:54:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.openstack.org/637955 | |
| 18:09:22 | melwitt | mriedem: omg, thank you. I was completely baffled by the 0'ed out instance uuid and could not imagine how that could happen | |
| 18:12:20 | melwitt | alex_xu: by unrelated, I meant that project_only=True isn't related to the instance target. the only thing that project_only=True does is enforce the request context project, not the instance project. so I was thinking that changing to use instance target is an unrelated change from removing project_only=True. if we remove project_only=True without changing instance target, we get the same behavior as today except that users choose to | |
| 18:12:20 | melwitt | enforce request context project via policy instead of hard-coded | |
| 18:16:36 | efried | aspiers: Still around? | |
| 18:20:31 | aspiers | efried: just got back | |
| 18:20:37 | efried | aspiers: Sorry that took a little longer than expected (interrupted by internal work, among other things), but I got it figured out. Now I have to go around and fix all the test cases that call into the shimmed API. | |
| 18:20:46 | aspiers | efried: nice work :) | |
| 18:20:54 | efried | WIP incoming, lemme finish writing up the commit message. | |
| 18:20:56 | aspiers | I bet whatever you changed would have taken me a week | |
| 18:21:26 | aspiers | mriedem: OK thanks, I'll do that | |
| 18:21:32 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: De-shim PlacementFixture https://review.openstack.org/637959 | |
| 18:21:41 | efried | aspiers: ^ | |
| 18:21:43 | aspiers | mriedem, melwitt: is it OK to reference 2 blueprints in a single commit message? | |
| 18:21:46 | aspiers | efried: ack | |
| 18:22:05 | efried | I originally wrote it on top of your patch, to make sure your test case worked, and it did. | |
| 18:22:15 | efried | But this version will be lacking your delete helper. | |
| 18:22:27 | efried | Once I've got it polished I'll rebase it under yours. | |
| 18:22:34 | efried | or you can | |
| 18:22:40 | aspiers | Sure, I can do that | |
| 18:22:48 | aspiers | I have some other tweaks to go in mine anyway | |
| 18:23:09 | melwitt | aspiers: I don't see why not. doing that would just put the link on both lp whiteboard AFAIK | |
| 18:23:19 | aspiers | not to mention fixes for the glaring errors you and mriedem are about to uncover ;-) | |
| 18:23:31 | aspiers | melwitt: cool thanks, that's what I expected | |
| 18:25:19 | aspiers | woohoo, my version switcher patch landed on https://docs.openstack.org/ | |
| 18:25:36 | aspiers | hopefully it doesn't break anything | |
| 18:25:43 | aspiers | s/version/release/ | |
| 18:40:24 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Fallback to the first element if no nodes match https://review.openstack.org/637965 | |
| 19:19:54 | openstackgerrit | Merged openstack/nova master: api-ref: warn about changing/unsetting AZ name with instances https://review.openstack.org/635315 | |
| 19:29:41 | openstackgerrit | melanie witt proposed openstack/nova master: Add user_id field to InstanceMapping https://review.openstack.org/633350 | |
| 19:29:41 | openstackgerrit | melanie witt proposed openstack/nova master: Add online data migration for populating user_id https://review.openstack.org/633351 | |
| 19:35:24 | openstackgerrit | Merged openstack/nova master: api-ref: mention policy defaults for aggregates https://review.openstack.org/635317 | |
| 19:37:10 | efried | mriedem: easy +A if you please https://review.openstack.org/#/c/637657/ | |
| 19:45:43 | mriedem | +W | |
| 19:50:11 | efried | thanks mriedem | |
| 19:57:42 | efried | aspiers: I had to give up. Your fix is gateward. | |
| 20:28:34 | openstackgerrit | Merged openstack/nova master: Fix fake DELETE in PlacementFixture https://review.openstack.org/637657 | |
| 20:42:25 | openstackgerrit | Merged openstack/os-vif master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610068 | |
| 21:07:56 | mriedem | i wonder how many operators are trying to recover an instance from a failed resize confirm/revert where the instance is in ERROR status and they fixed something in the compute and just want to retry the confirm/revert operation, but can't b/c the API doesn't allow those operations on an instance in ERROR status. seems weird that the os-resetStatus server action API doesn't allow you to put an instance in 'resized' status agai | |
| 21:07:56 | mriedem | rom error) | |
| 21:08:10 | mriedem | dansmith: ever thunk on that?^ | |
| 21:08:48 | dansmith | well, it would be pretty bad if you put it into resized status when it wasn't, | |
| 21:08:58 | dansmith | and it'd be hard to tell when and when not to do something like that | |
| 21:09:29 | mriedem | if you got through revert_resize on the dest but finish_revert_resize fails on the source, you can maybe hard reboot your way out of it | |
| 21:09:41 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Don't reference unassigned variable in xml matchers in test https://review.openstack.org/637965 | |
| 21:09:42 | mriedem | if revert_resize on the dest fails, you're kind of stuck there now | |
| 21:09:48 | mriedem | anywho | |
| 21:10:06 | dansmith | right but is it realistic for an operator to know that it failed on finish_revert_resize? | |
| 21:10:09 | dansmith | even if they fix something, | |
| 21:10:28 | dansmith | they could reset into resized state from another place where they think it's likely to help and further break stuff | |
| 21:10:42 | dansmith | I mean, I know what you're saying, but.. it's a dangerous tool | |
| 21:10:56 | mriedem | sure, and i wouldn't expect most operators to know that unless they were pretty knowledgeable on low level nova | |
| 21:11:06 | mriedem | which their time is probably just better spent on rebuilding the vm | |
| 21:11:16 | artom | And, is it needed? I mean, if they have data they want to keep on a volume, they can just attach that to a new instance with the correct flavor? | |
| 21:11:18 | mriedem | and writing an apology to the user | |
| 21:11:51 | mriedem | artom: i'm thinking more about local storage, no volumes | |
| 21:12:08 | artom | So actually, is a resize destructive in that respect? | |
| 21:12:09 | mriedem | granted your data should be stored on a persistent volume | |
| 21:12:31 | mriedem | no, the local disks are left intact on the source host when you resize | |
| 21:12:46 | mriedem | they are destroyed when you confirm the resize to keep the instance on the dest host | |
| 21:12:53 | mriedem | and re-used on the source if you revert | |
| 21:13:48 | artom | And your idea is - if the resize went to ERROR, rather than just reverting back to the source, allow them to force completion, so to speak? | |
| 21:14:00 | mriedem | not force completion, retry | |
| 21:14:12 | mriedem | but, | |
| 21:14:20 | mriedem | that retry could fail if we got part way the first time | |
| 21:14:25 | mriedem | so retrying isn't without risk either | |
| 21:14:42 | mriedem | i.e. revert_resize on dest fails the 2nd time b/c we already deleted some stuff the first time | |
| 21:15:01 | artom | Yeah... | |
| 21:15:31 | artom | In that situation, what's wrong with going through the resize again, after reverting the failed attempt? | |
| 21:15:33 | mriedem | i've just got some sticky logic in the cross-cell revert code where if something fails part way through it's not so good | |
| 21:15:54 | mriedem | "reverting the failed attempt"? | |
| 21:16:02 | mriedem | s/reverting/fixing/? | |
| 21:16:23 | mnaser | maybe in an ideal world we can collect more of the cases where finish_revert_resize fails and test those out *before* starting to revert the resize | |
| 21:16:24 | artom | Oh. You can't revert out of an ERROR, can you? | |
| 21:16:25 | mriedem | you could reset status to active and try another resize | |
| 21:16:41 | mriedem | no you can only confirm/revert a server in 'resized' status | |
| 21:16:43 | mnaser | no you cant revert out of ERROR, only RESIZE_CONFIRM i think | |