Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-20
15:00:00 dansmith mriedem: and.. can you even roll it back? often times block device resizes are one-way
15:00:06 mriedem alternatively, if cinder has a policy rule for extending an attached volume, then the admin can disable this from the cinder side until they know they can handle it
15:00:25 mriedem mgagne: does the cinder side of this have a unique policy rule for extending attached volumes?
15:00:34 mriedem or is it just the same as os-extend for 'available' non-attached volumes?
15:01:16 jaypipes knikolla_phone: here's a dragon for you: http://bit.ly/2sLVKtR
15:01:28 mriedem mgagne: because if i were a xen or vmware or hyperv shop, and this just will never work, i'd want to disable the cinder api behavior globally
15:02:28 knikolla_phone jaypipes: the nova-manage project quota_refresh_usage command didn't do anything.
15:02:56 knikolla_phone Newton -> ocata was the only problematic upgrade so far for us. :(
15:04:29 jaypipes knikolla_phone: other than manually going through your DB, I'm not entirely sure I can help you on this channel. There's some mismatch somewhere between the project_quota_usages or quota_usages table and the "true" resource tables like instances, instance_extra, pci_devices, etc
15:04:54 jaypipes knikolla_phone: is this isolated to a particular tenant? or general?
15:06:01 knikolla_phone jaypipes: no, that's okay, you've helped more than enough. This is only limited to one tenant.
15:06:27 jaypipes knikolla_phone: and only for their instances quota?
15:07:32 knikolla_phone jaypipes: whatever resources two of the instances we manually deleted used (instances, vcpu, ram, etc)
15:08:01 jaypipes hmm
15:08:37 mriedem dansmith: ok, so i think policy blocker on the cinder side is key, i've put that in the cinder change https://review.openstack.org/#/c/454287/
15:08:42 jaypipes knikolla_phone: but does the instances usage show correct? meaning, does it show the tenant using instances that were deleted?
15:08:50 dansmith mriedem: ack
15:08:50 mriedem dansmith: and then we just report the shit out of when it fails on the nova compute side in pike
15:08:58 mriedem using action events and faults
15:11:15 knikolla_phone jaypipes: yes. The usage is correct to what it would be if those two instances weren't deleted (which is weird because we manually deleted five of them). Quota_usages table reflects that amount, and updating it to random values and then using quota_refresh_usage command, it preserves the random amount.
15:13:10 jaypipes knikolla_phone: what's in the reservations table for that project?
15:21:36 knikolla_phone jaypipes: what does uuid in that table (reservations) map to? Is it an instance uuid?
15:21:56 jaypipes knikolla_phone: yep
15:23:18 knikolla_phone jaypipes: okay, i see a deleted row. Going to try setting that too.
15:23:32 knikolla_phone Thanks for all your help.
15:23:36 mnaser i've been doing some digging around and i cannot seem to find where root_device_name gets assigned for vms.. can someone give a hint :(
15:23:42 jaypipes knikolla_phone: hey, no prob :)
15:23:53 mnaser i've discovered a bug with when using virtio-scsi with nova
15:24:06 jaypipes mnaser: depends on the hypervisor.
15:24:13 jaypipes mnaser: you talking about libvirt?
15:24:15 mnaser root_device_name gets set to /dev/vda .. but when attaching cinder volumes, it tries to attach them at /dev/sdX, and tries to assign the same pci address
15:24:16 mnaser yep
15:24:38 jaypipes mnaser: hmm...
15:24:49 jaypipes mnaser: might need to bring in mdbooth on that one.
15:24:51 mnaser i turned off the vm, attached the cinder block device and then started it
15:24:55 jaypipes mdbooth: Matt, any ideas?
15:24:57 mnaser and it actually generated invalid libvirt xml
15:25:01 mnaser i have a copy of it, let me paste
15:26:04 mnaser well looks like paste.openstack.org isn't loading
15:26:11 mriedem mnaser: _validate_and_build_base_options in nova.compute.api.API
15:26:26 mnaser https://gist.github.com/mnaser/8b000d20590e4343e4d7a73459d85210
15:26:31 mriedem sets it in the base_options dict which eventually gets set on the instance (really the BuildRequest now)
15:26:42 mnaser notice the address of the first 2 disks
15:26:44 mnaser is the same
15:26:47 mnaser so qemu never boots the vm
15:27:29 mnaser i added a LOG.error(disk_info) in the libvirt tooling
15:27:30 mnaser and this is what came out of it too - https://gist.github.com/mnaser/aedd44b9df018786a144f7c719ec5a13
15:27:49 mnaser notice how 'dev': 'vda' (which is ephermal) and the 'dev': 'sda' (which is attached cinder block device)
15:28:11 mnaser i am guessing that's where the issue is stemming, the first one should be sda too (because using virtio-scsi with the image hw_ parameters)
15:28:12 mriedem mnaser: https://github.com/openstack/nova/blob/9045a98145f0834446d668e0282ddb1c6f2d136c/nova/compute/api.py#L921 is eventually what gets set in the instance.root_device_name field
15:29:22 mnaser i've confirmed with two different installs that booting an image with hw_scsi_model=virtio-scsi and hw_disk_bus=scsi will work but no cinder volumes will successfully attach: libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Duplicate ID 'drive-scsi0-0-0-0' for drive
15:29:30 mdbooth mnaser: what's the symptom of the weird xml?
15:29:35 mdbooth Does it barf?
15:29:50 mnaser so, if you try to attach it to an already running vm, you get the libvirt error above (duplicate id)
15:30:19 mnaser if you turn the vm off, attach (really a no-op if its offline), start it, it'll generate that invalid xml and fail
15:30:32 mnaser so im not sure if libvirt or nova is to blame in putting the duplicate address ther
15:30:45 mdbooth We really shouldn't be looking at root_device_name
15:31:34 mnaser i've just been kinda trying to understand the code and i dont know if i can point the finger at nova or libvirt yet
15:32:03 mnaser but it's obviously just an issue when using virtio-scsi
15:37:48 mgagne mriedem: supports_volume_extend vs supports_extend_volume ? RE https://review.openstack.org/#/c/454322/18/nova/virt/driver.py
15:38:31 openstackgerrit Dan Smith proposed openstack/nova master: Fix using targeted cell context when finding services in cells https://review.openstack.org/475830
15:38:53 dansmith dtantsur: mriedem ^
15:39:01 dtantsur nice
15:40:09 openstackgerrit Kaitlin Farr proposed openstack/nova master: Remove deprecated keymgr code https://review.openstack.org/439855
15:41:04 mriedem melwitt: i think you might have gotten delete happy in the tests here https://review.openstack.org/#/c/457862/
15:41:33 mriedem mgagne: i'm ok with either - i suppose supports_extend_volume to match the driver's method name
15:41:49 mgagne that's my opinion
15:50:22 mriedem mdbooth: i think that root_device_name thing is a legacy bdm v1 thing, but not totally sure
15:50:44 mriedem some kind of ec2 carry over
15:51:36 mdbooth mriedem: I haven't followed the code all the way down, but I suspect we could ignore it
15:54:36 openstackgerrit Jay Pipes proposed openstack/nova master: placement: scheduler uses allocation candidates https://review.openstack.org/475600
15:56:50 dansmith jaypipes: quick +W bait: https://review.openstack.org/#/c/475830/
15:57:01 dansmith jaypipes: fixes the ironic people
15:57:31 jaypipes dansmith: wallaby'd
15:57:38 dansmith thanks
15:57:41 dtantsur thanks all!
15:57:51 dansmith I guess cdent still isn't around today?
15:58:06 jaypipes dansmith: he's on PTO for next two weeksd.
15:58:11 dansmith oh my
15:58:20 jaypipes dansmith: after which he will be a VMWare employee.
15:59:29 dansmith oh
15:59:32 dansmith my
15:59:35 dansmith GAWD
16:01:34 openstackgerrit Merged openstack/nova master: Count networks to check quota https://review.openstack.org/446241
16:02:29 gibi_ jaypipes: good luck with the api.fault bug
16:02:46 jaypipes gibi_: yeah, I might need to pass on that :(
16:02:56 jaypipes gibi_: that's some old-ass code in there :)
16:03:13 gibi_ jaypipes: I hope somebody can say 'delete that code!'
16:04:47 jaypipes gibi_: it's possible. some day. https://twitter.com/dims/status/542002018970529792
16:06:27 gibi_ :)
16:07:36 openstackgerrit Merged openstack/nova master: Add SecurityGroupList.get_counts() https://review.openstack.org/446245
16:08:08 openstackgerrit Merged openstack/nova master: Count security groups to check quota https://review.openstack.org/457860
16:09:12 openstackgerrit Merged openstack/nova master: Add FixedIPList.get_count_by_project() https://review.openstack.org/446246
16:09:48 openstackgerrit Merged openstack/nova master: Count fixed ips to check quota https://review.openstack.org/457861
16:10:17 openstackgerrit Merged openstack/nova master: Add FloatingIPList.get_count_by_project() https://review.openstack.org/446247
16:10:23 dansmith wow, just count those counting quotas merges
16:21:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Factor out duplicated notification sample data (3) https://review.openstack.org/452820
16:21:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Factor out duplicated notification sample data https://review.openstack.org/452818
16:21:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: Factor out duplicated notification sample data (2) https://review.openstack.org/452819
16:21:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: Factor out duplicated notification sample data https://review.openstack.org/475860

Earlier   Later