| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-19 | |||
| 17:05:04 | mriedem | those could be stable branches, checking logstash | |
| 17:05:57 | smcginnis | Grenade on the first one, so maybe? | |
| 17:06:04 | mriedem | sean-k-mooney: you hit a nova bug http://logs.openstack.org/15/620115/23/check/tempest-full/fcabb4b/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Feb_19_14_49_49_277325 | |
| 17:06:17 | mriedem | http://status.openstack.org/elastic-recheck/#1770211 | |
| 17:06:25 | mriedem | WARNING nova.virt.block_device [None req-2c3d004a-05e7-43d3-af5e-ccac75bed5fd tempest-AttachVolumeNegativeTest-1375016298 tempest-AttachVolumeNegativeTest-1375016298] [instance: 4b50efcf-0d26-4b01-91f0-1b8b1e9eef0f] Guest refused to detach volume aea37477-6c0c-4727-8092-da3b3d2f7c5c: DeviceDetachFailed: Device detach failed for vdb: Unable to detach the device from the live config. | |
| 17:06:45 | mriedem | i remember cfriesen asking about this still being an issue the other day | |
| 17:07:08 | mriedem | smcginnis: yeah looks like it's hitting on grenade so it's just a stable branch issue with devstack at this poin, | |
| 17:07:09 | mriedem | *point | |
| 17:07:10 | dansmith | mriedem: I saw a ping from you fly by from last night.. can you link me again so I don't have to dig out mah logs? | |
| 17:07:16 | mriedem | i'll work on backporting those devstack / etcd changes | |
| 17:07:33 | mriedem | dansmith: https://review.openstack.org/#/c/635315/1 | |
| 17:10:20 | dansmith | mriedem: nice | |
| 17:11:18 | cfriesen | mriedem: smcginnis: I was hitting this:https://bugs.launchpad.net/nova/+bug/1815949 | |
| 17:11:19 | openstack | Launchpad bug 1815949 in OpenStack Compute (nova) "missing special-case libvirt exception during device detach" [Medium,Confirmed] | |
| 17:11:39 | smcginnis | Hmm, that's a new one to me. | |
| 17:12:02 | sean-k-mooney | ok ya that makes sense. im pretty sure there are other voluem attachemtn failures like https://bugs.launchpad.net/cinder/+bug/1814916 too which are unrelated to tooz | |
| 17:12:03 | mriedem | cfriesen: ok that looks different than the 'unable to detach from transient / live config' | |
| 17:12:04 | openstack | Launchpad bug 1814916 in Cinder "MessagingTimeout in cinder-api during volume attachment update due to VolumeAttachmentNotFound in cinder-volume" [Medium,Confirmed] | |
| 17:12:19 | mriedem | yeah i don't know what's up with that volume attachment not found one | |
| 17:12:31 | mriedem | the api finds the record and then c-vol blows up b/c it's gone from the db, so clearly a race somewhere | |
| 17:12:57 | cfriesen | mriedem: smcginnis: but after fixing that I'm hitting the "Guest refused to detach" issue. I'm guessing it's related to stalls we're seeing in the guest, but still digging into causes. | |
| 17:12:57 | mriedem | probably just need someone to trace requests in a log to see when the PUT request is made and when the DELETE request is made | |
| 17:14:26 | openstackgerrit | Adam Spiers proposed openstack/nova master: WIP: Convert driver supported capabilities to compute node provider traits https://review.openstack.org/538498 | |
| 17:15:11 | aspiers | mriedem, efried: I've just finished the first version of the tests which were missing for capabilities to traits: https://review.openstack.org/#/c/538498/12/nova/tests/functional/test_servers.py | |
| 17:15:28 | mriedem | etcd changes for devstack backported https://review.openstack.org/#/q/topic:bug/1810526+(status:open+OR+status:merged) | |
| 17:15:43 | aspiers | mriedem, efried: I'm sure you'll find plenty of things to improve, but hopefully this is a good base | |
| 17:16:21 | aspiers | not sure if I went overkill on the tests for standard vs. custom traits, and capabilities vs. update_provider_tree | |
| 17:22:42 | aspiers | mriedem: BTW I kept you as the author and added myself via Co-Authored-By - was that the right thing to do? | |
| 17:22:51 | aspiers | not sure what the convention is here | |
| 17:24:18 | mriedem | aspiers: yup | |
| 17:24:28 | mriedem | more testing > less testing | |
| 17:24:50 | aspiers | mriedem: also, should I put "related to blueprint expose-host-capabilities" in the commit message? | |
| 17:32:48 | mriedem | hmm, that's not approved https://blueprints.launchpad.net/nova/+spec/expose-host-capabilities but the change is essentially trying to solve that same problem | |
| 17:32:50 | mriedem | so sure | |
| 17:33:26 | mriedem | the spec for that bp predates traits https://review.openstack.org/#/c/222200/ | |
| 17:33:35 | mriedem | so it was trying to solve it with something different | |
| 17:43:32 | sean-k-mooney | adrianc: o/ are you wokring on a new version of https://review.openstack.org/#/c/620115/ | |
| 17:44:05 | sean-k-mooney | adrianc: if not i might start adressing stephenfin nits after i get something to eat. | |
| 17:54:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance to make place for port healing https://review.openstack.org/637953 | |
| 17:54:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.openstack.org/637955 | |
| 17:54:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance (2) https://review.openstack.org/637954 | |
| 18:09:22 | melwitt | mriedem: omg, thank you. I was completely baffled by the 0'ed out instance uuid and could not imagine how that could happen | |
| 18:12:20 | melwitt | enforce request context project via policy instead of hard-coded | |
| 18:12:20 | melwitt | alex_xu: by unrelated, I meant that project_only=True isn't related to the instance target. the only thing that project_only=True does is enforce the request context project, not the instance project. so I was thinking that changing to use instance target is an unrelated change from removing project_only=True. if we remove project_only=True without changing instance target, we get the same behavior as today except that users choose to | |
| 18:16:36 | efried | aspiers: Still around? | |
| 18:20:31 | aspiers | efried: just got back | |
| 18:20:37 | efried | aspiers: Sorry that took a little longer than expected (interrupted by internal work, among other things), but I got it figured out. Now I have to go around and fix all the test cases that call into the shimmed API. | |
| 18:20:46 | aspiers | efried: nice work :) | |
| 18:20:54 | efried | WIP incoming, lemme finish writing up the commit message. | |
| 18:20:56 | aspiers | I bet whatever you changed would have taken me a week | |
| 18:21:26 | aspiers | mriedem: OK thanks, I'll do that | |
| 18:21:32 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: De-shim PlacementFixture https://review.openstack.org/637959 | |
| 18:21:41 | efried | aspiers: ^ | |
| 18:21:43 | aspiers | mriedem, melwitt: is it OK to reference 2 blueprints in a single commit message? | |
| 18:21:46 | aspiers | efried: ack | |
| 18:22:05 | efried | I originally wrote it on top of your patch, to make sure your test case worked, and it did. | |
| 18:22:15 | efried | But this version will be lacking your delete helper. | |
| 18:22:27 | efried | Once I've got it polished I'll rebase it under yours. | |
| 18:22:34 | efried | or you can | |
| 18:22:40 | aspiers | Sure, I can do that | |
| 18:22:48 | aspiers | I have some other tweaks to go in mine anyway | |
| 18:23:09 | melwitt | aspiers: I don't see why not. doing that would just put the link on both lp whiteboard AFAIK | |
| 18:23:19 | aspiers | not to mention fixes for the glaring errors you and mriedem are about to uncover ;-) | |
| 18:23:31 | aspiers | melwitt: cool thanks, that's what I expected | |
| 18:25:19 | aspiers | woohoo, my version switcher patch landed on https://docs.openstack.org/ | |
| 18:25:36 | aspiers | hopefully it doesn't break anything | |
| 18:25:43 | aspiers | s/version/release/ | |
| 18:40:24 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Fallback to the first element if no nodes match https://review.openstack.org/637965 | |
| 19:19:54 | openstackgerrit | Merged openstack/nova master: api-ref: warn about changing/unsetting AZ name with instances https://review.openstack.org/635315 | |
| 19:29:41 | openstackgerrit | melanie witt proposed openstack/nova master: Add online data migration for populating user_id https://review.openstack.org/633351 | |
| 19:29:41 | openstackgerrit | melanie witt proposed openstack/nova master: Add user_id field to InstanceMapping https://review.openstack.org/633350 | |
| 19:35:24 | openstackgerrit | Merged openstack/nova master: api-ref: mention policy defaults for aggregates https://review.openstack.org/635317 | |
| 19:37:10 | efried | mriedem: easy +A if you please https://review.openstack.org/#/c/637657/ | |
| 19:45:43 | mriedem | +W | |
| 19:50:11 | efried | thanks mriedem | |
| 19:57:42 | efried | aspiers: I had to give up. Your fix is gateward. | |
| 20:28:34 | openstackgerrit | Merged openstack/nova master: Fix fake DELETE in PlacementFixture https://review.openstack.org/637657 | |
| 20:42:25 | openstackgerrit | Merged openstack/os-vif master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610068 | |
| 21:07:56 | mriedem | rom error) | |
| 21:07:56 | mriedem | i wonder how many operators are trying to recover an instance from a failed resize confirm/revert where the instance is in ERROR status and they fixed something in the compute and just want to retry the confirm/revert operation, but can't b/c the API doesn't allow those operations on an instance in ERROR status. seems weird that the os-resetStatus server action API doesn't allow you to put an instance in 'resized' status agai | |
| 21:08:10 | mriedem | dansmith: ever thunk on that?^ | |
| 21:08:48 | dansmith | well, it would be pretty bad if you put it into resized status when it wasn't, | |
| 21:08:58 | dansmith | and it'd be hard to tell when and when not to do something like that | |
| 21:09:29 | mriedem | if you got through revert_resize on the dest but finish_revert_resize fails on the source, you can maybe hard reboot your way out of it | |
| 21:09:41 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Don't reference unassigned variable in xml matchers in test https://review.openstack.org/637965 | |
| 21:09:42 | mriedem | if revert_resize on the dest fails, you're kind of stuck there now | |
| 21:09:48 | mriedem | anywho | |
| 21:10:06 | dansmith | right but is it realistic for an operator to know that it failed on finish_revert_resize? | |
| 21:10:09 | dansmith | even if they fix something, | |
| 21:10:28 | dansmith | they could reset into resized state from another place where they think it's likely to help and further break stuff | |
| 21:10:42 | dansmith | I mean, I know what you're saying, but.. it's a dangerous tool | |
| 21:10:56 | mriedem | sure, and i wouldn't expect most operators to know that unless they were pretty knowledgeable on low level nova | |
| 21:11:06 | mriedem | which their time is probably just better spent on rebuilding the vm | |
| 21:11:16 | artom | And, is it needed? I mean, if they have data they want to keep on a volume, they can just attach that to a new instance with the correct flavor? | |
| 21:11:18 | mriedem | and writing an apology to the user | |
| 21:11:51 | mriedem | artom: i'm thinking more about local storage, no volumes | |
| 21:12:08 | artom | So actually, is a resize destructive in that respect? | |
| 21:12:09 | mriedem | granted your data should be stored on a persistent volume | |
| 21:12:31 | mriedem | no, the local disks are left intact on the source host when you resize | |
| 21:12:46 | mriedem | they are destroyed when you confirm the resize to keep the instance on the dest host | |