Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-21
22:53:16 mriedem it's a copy of the initial server create request
22:53:19 cfriesen mriedem: don't we feed the request spec to the scheduler when searching for a dest?
22:53:26 mriedem minus ports and bdms
22:53:38 mriedem cfriesen: sure do
22:53:42 mriedem sometimes slightly modified
22:53:58 mriedem so my guess would be, people have out of tree filters,
22:54:06 mriedem and those filters need to know information about volumes and ports on the instance,
22:54:13 mriedem and they want to get it from the request spec,
22:54:20 cfriesen I was going to ask about ports/bdms, but then you pointed out that they were subtracted anyways.
22:54:23 mriedem because they are too lazy to hit the DB, or cinder or neutron APIs
22:54:37 mriedem they aren't subtracted, they just aren't persisted in the request spec
22:54:43 sean-k-mooney[m] mriedem: well if we send the request spec to the schduler when livemigrating then would you not want it to have the list of networks you vm currently has instead of the inital set
22:55:08 mriedem sean-k-mooney[m]: we don't have any filters that look at that
22:55:35 mriedem "nova doesn't provide the thing my private out of tree filter needs" isn't a bug
22:56:14 sean-k-mooney[m] the pci passthrough fileter looks at this
22:56:37 sean-k-mooney[m] that said i guess we dont supprot hot attach for sriov interfaces currently
22:56:54 sean-k-mooney[m] you can detach actully but just not attach
22:57:45 sean-k-mooney[m] actully the pci passthroguh filter looks at the pci requests specs which i guess is a little different
22:57:50 mriedem right
22:58:38 mriedem now i'm sure it's possible to attach a port to an instance where the network that port is on is available to the current compute host, and then live migrate the instance to another host where that network is't available,
22:58:43 mriedem but that's also a problem we have with server create today
22:59:28 sean-k-mooney[m] i always forget how the routed networks stuff works but do they just track the subnets in placement or is there also a sechuler filter
22:59:42 mriedem there isn't a filter
22:59:46 mriedem they use aggregates somehow
22:59:52 mriedem i still don't know how it actually works
23:00:00 sean-k-mooney[m] mriedem: on boot that is "fine" because we can retry. on livemigrate not so much
23:00:02 mriedem without doing the port stuff in conductor
23:03:01 sean-k-mooney[m] well if the request spec is intended to store teh inital request then ya thats not a bug
23:07:15 sean-k-mooney[m] mriedem: by the way was i chatting to you at the ptg about the pci white list parser bug i found. i think i tracked it to here https://github.com/openstack/nova/blob/master/nova/pci/whitelist.py#L58 do you know if there is a reason we use jsonutils directly here and not oslo.config
23:10:55 mriedem sean-k-mooney[m]: wasn't me
23:12:06 sean-k-mooney[m] not looking to blame :)
23:13:45 sean-k-mooney[m] the call to jsonutils.loads(jsonspec) can raise error other then value error so if you have unicode in your whitelist the nova compute agent can die because the exception is not caught
23:14:05 mriedem lyarwood was looking at something similar before the ptg
23:15:02 sean-k-mooney[m] mriedem: i hard locked 15 server with this bug + a docker/centos kernel bug the week before the ptg
23:15:32 sean-k-mooney[m] our lab time was interested in why our rack was suddenly draw 5% of the total phase
23:15:54 mriedem efried: this is a fun one up your ksa alley https://bugs.launchpad.net/nova/+bug/1752152
23:15:56 openstack Launchpad bug 1752152 in OpenStack Compute (nova) queens "Attach Volume Fails with secure call to cinder" [Undecided,Triaged]
23:17:33 sean-k-mooney[m] any way i was talking to dug helmen about if i should "fix" it in oslo or nova ill proably submit a patch to both and see which one merges first
23:18:28 efried mriedem: Looks like it has an owner?
23:20:48 mriedem efried: dikonoor doesn't seem to actually be working on it
23:21:03 mriedem maybe run that through the internal powervc sametime channel :)
23:21:14 mriedem er verse
23:21:16 mriedem er slack
23:21:35 efried mriedem: Do we need to use https to get the version document??
23:22:17 mriedem i assume https is what's in the service catalog?
23:23:13 mriedem apparently you don't need a token to get the version document, which is ok with http
23:23:46 mriedem so uh,
23:23:59 mriedem how terrible would it be if we s/https/http/ in this cinderclient code?
23:24:20 efried MITM?
23:24:29 mriedem aaS
23:25:44 efried mordred: Care to render an opinion? (TL;DR: is it okay to demote https to http if we're just getting the version document?)
23:26:15 cfriesen any way to do a MiTM attack with the version document?
23:26:29 cfriesen whoops, efried already said that
23:26:34 efried Other than, like, corrupting it.
23:26:47 efried Or maybe spoofing a microversion with a known security flaw?
23:26:54 cfriesen yeah, that's what I was thinking
23:28:14 mriedem otherwise i can probably hack something like where we actually create a cinderclient client object, and then use it's internal client to make the request
23:28:16 efried mriedem: The alternative, though, is for this method (still in cinderclient) to use proper ksa loading instead of direct requests.get
23:28:18 mriedem or, just use ksa
23:28:23 efried yeah, any of that.
23:31:16 efried mriedem: If we want to keep that method in cinderclient, we could add a kwarg that lets you pass in a ksa session. Then from nova use the _SESSION global which we've already loaded by that point.
23:31:17 openstackgerrit Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081
23:31:25 efried mriedem: Howzat sound?
23:31:37 mriedem efried: i'd then have to plumb that through into cinderclient i think
23:31:44 efried yes, that's what I mean.
23:31:50 mriedem see https://bugs.launchpad.net/nova/+bug/1752152/comments/2
23:31:51 openstack Launchpad bug 1752152 in OpenStack Compute (nova) queens "Attach Volume Fails with secure call to cinder" [Undecided,Triaged]
23:31:57 mriedem i would like to just cut out cinderclient altogether
23:31:58 efried The alternative is duplicating the cinderclient code in nova.
23:32:03 efried yeah, that.
23:32:07 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1746509 https://review.openstack.org/555092
23:32:08 openstackgerrit melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093
23:32:08 openstack bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,Confirmed] https://launchpad.net/bugs/1746509
23:32:51 mriedem also, we can't change cinderclient API code and require a new minimum version on stable
23:32:55 mriedem so that's kind of a non-starter
23:34:01 mriedem alright, i can make that my tomorrow if i can get a recreate with devstack configuring cinder for ssl
23:37:35 mriedem actually this is strange because devstack already sets up the endpoint using https if tls-proxy is enabled, which it is in our CI runs
23:40:05 melwitt mriedem: not always right? we had to enable it explicitly for nova-next, right?
23:40:22 mriedem http://logs.openstack.org/45/508345/13/check/tempest-full/893771e/controller/logs/devstacklog.txt.gz#_2018-03-15_19_48_10_167
23:40:28 openstackgerrit melanie witt proposed openstack/nova master: Add functional regression test for bug 1746509 https://review.openstack.org/555092
23:40:29 mriedem ENABLED_SERVICES=g-reg,rabbit,n-api,c-api,g-api,mysql,tempest,etcd3,s-proxy,q-dhcp,n-api-meta,tls-proxy,q-l3,c-sch,n-novnc,s-object,peakmem_tracker,n-cauth,q-metering,key,n-cond,s-container,q-meta,q-svc,placement-api,n-cpu,s-account,c-vol,n-obj,c-bak,q-agt,cinder,n-sch,dstat
23:40:29 openstackgerrit melanie witt proposed openstack/nova master: Move _make_instance_list call outside of DB transaction context https://review.openstack.org/555093
23:40:29 openstack bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] https://launchpad.net/bugs/1746509 - Assigned to melanie witt (melwitt)
23:42:00 melwitt hm, okay
23:42:35 mriedem maybe it's a legacy job vs zuulv3 job thing, not sure
23:42:38 mriedem tempest-full is zuulv3 native
23:42:41 melwitt yep, enabled there. not sure why we had to enable it in nova-next
23:45:38 mriedem yeah right here http://logs.openstack.org/45/508345/13/check/nova-next/aa61d86/logs/screen-c-api.txt.gz#_Mar_15_20_00_06_201395
23:45:43 mriedem Mar 15 20:00:06.201395 ubuntu-xenial-inap-mtl01-0002993032 devstack@c-api.service[32240]: INFO cinder.api.openstack.wsgi [req-2e3a55fd-3ed7-4b7f-b902-c4671afa83c4 req-d6e11ad6-95b0-42ed-bcb2-af3996a69096 admin admin] GET https://198.72.124.205/volume/
23:46:11 mriedem nstack.wsgi [req-2e3a55fd-3ed7-4b7f-b902-c4671afa83c4 req-d6e11ad6-95b0-42ed-bcb2-af3996a69096 admin admin] https://198.72.124.205/volume/ returned with HTTP 300
23:46:11 mriedem pi.openstack.wsgi [req-2e3a55fd-3ed7-4b7f-b902-c4671afa83c4 req-d6e11ad6-95b0-42ed-bcb2-af3996a69096 admin admin] Calling method '<bound method VersionsController.all of <cinder.api.versions.VersionsController object at 0x7f81c53a0550>>' {{(pid=32243) _process_stack /opt/stack/new/cinder/cinder/api/openstack/wsgi.py:872}} Mar 15 20:00:06.202926 ubuntu-xenial-inap-mtl01-0002993032 devstack@c-api.service[32240]: INFO cinder.api
23:46:11 mriedem Mar 15 20:00:06.201628 ubuntu-xenial-inap-mtl01-0002993032 devstack@c-api.service[32240]: DEBUG cinder.api.openstack.wsgi [req-2e3a55fd-3ed7-4b7f-b902-c4671afa83c4 req-d6e11ad6-95b0-42ed-bcb2-af3996a69096 admin admin] Empty body provided in request {{(pid=32243) get_body /opt/stack/new/cinder/cinder/api/openstack/wsgi.py:718}} Mar 15 20:00:06.202003 ubuntu-xenial-inap-mtl01-0002993032 devstack@c-api.service[32240]: DEBUG cind
23:46:15 mriedem yikes
23:46:22 mriedem Mar 15 20:00:06.202003 ubuntu-xenial-inap-mtl01-0002993032 devstack@c-api.service[32240]: DEBUG cinder.api.openstack.wsgi [req-2e3a55fd-3ed7-4b7f-b902-c4671afa83c4 req-d6e11ad6-95b0-42ed-bcb2-af3996a69096 admin admin] Calling method '<bound method VersionsController.all of <cinder.api.versions.VersionsController object at 0x7f81c53a0550>>' {{(pid=32243) _process_stack /opt/stack/new/cinder/cinder/api/openstack/wsgi.py:872}}
23:46:26 mriedem that's hitting the versions controller
23:46:30 mriedem Mar 15 20:00:06.202926 ubuntu-xenial-inap-mtl01-0002993032 devstack@c-api.service[32240]: INFO cinder.api.openstack.wsgi [req-2e3a55fd-3ed7-4b7f-b902-c4671afa83c4 req-d6e11ad6-95b0-42ed-bcb2-af3996a69096 admin admin] https://198.72.124.205/volume/ returned with HTTP 300
23:47:44 mriedem idk, maybe that's via the client doing it's own version negotiation
23:48:49 mriedem also, if this has been regressed since pike, i'm pretty sure we would have heard more about this by now right?
23:48:56 mriedem i assume most clouds are using ssl
23:50:53 openstackgerrit Artom Lifshitz proposed openstack/nova-specs master: NUMA-aware live migration https://review.openstack.org/552722

Earlier   Later