Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-06
15:05:36 sean-k-mooney you said you tought i was using an old set of cybog pataches when you looked at the error i got on monday but that was the latest set that were up at that point
15:05:44 Sundar sean-k-mooney: Great, thanks
15:05:59 dansmith Sundar: no, this is a distributed system.. you should never make that assumption
15:06:17 mriosfer Sean: if host got two sockets , is recomended to "force" hw:numa_nodes=2 ? If vcpus are more 2 / 4 / 6 / 8 etc...
15:06:26 efried Sundar: If you wanted to trap those failures (you can use the superclass ClientException) and convert them to DeviceProfileError("Couldn't communicate with cyborg!") I think that would be okay.
15:06:42 dansmith Sundar: calls from nova to cyborg (and back) may be routed to different api endpoints on subsequent calls, in different states of broken, misconfigured, or transient upgrade
15:06:47 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Use neutronclient's port binding APIs https://review.opendev.org/706295
15:06:52 sean-k-mooney mriosfer: that depens on your workloads but i do for larger vms
15:06:54 dansmith efried: agreed
15:07:06 stephenfin kashyap: np, I figured it would just be easier do it myself real quick. No skin off my back
15:07:12 mriosfer Sean : larger VM from 8 vcpu?
15:07:15 mriosfer or more
15:07:25 sean-k-mooney for small vms of 4-8 CPUs i will often keep them with hw:numa_nodes=1
15:07:39 sean-k-mooney ya over 8 i tend to always split
15:07:48 openstackgerrit Brian Rosmaita proposed openstack/nova master: Do not inherit cinder_ image properties https://review.opendev.org/706298
15:07:49 sean-k-mooney to hw:numa_nodes=2
15:08:04 stephenfin dansmith: That neutron fix yesterday was incomplete, btw. ralonsoh and I worked out the fix this morning if you care to look again https://review.opendev.org/#/c/706213/
15:08:22 dansmith stephenfin: I'm reading now
15:08:26 stephenfin ta
15:08:52 sean-k-mooney but that is based on host core counts of 10-16 cores per socket. if you have 32 cores per socket then you might adjust what you consider big
15:08:57 mtreinish stephenfin: thanks for the reminder, I'll re-review that PR today or tomorrow
15:08:58 dansmith stephenfin: where is the code that actually uses this?
15:09:23 stephenfin dansmith: Just nova/api/openstack/compute/floating_ips.py
15:09:52 dansmith stephenfin: "port_details" is not in that file
15:10:00 dansmith do you mean it just passes through the structure to the api requester?
15:10:04 Sundar efried, dansmith: Got it. Yes, I would handle service failures in distributed systems in general. I asked because I don't see these specific checks everywhere e.g. https://github.com/openstack/nova/blob/master/nova/image/glance.py . But, I agree it is better to do it everywhere.
15:10:29 stephenfin dansmith: sure it is - line 40
15:11:10 mtreinish stephenfin: the only thing is we're in prep for 3.0.0 (which removes 2.7). So that PR probably won't get released until then which we're shooting for mid-March
15:11:30 dansmith stephenfin: % grep -c port_details nova/api/openstack/compute/floating_ips.py
15:11:31 dansmith 0
15:12:11 stephenfin dansmith: It's definitely there :) https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/floating_ips.py#L40
15:12:34 dansmith oh is this a recent change? /me rebases
15:13:01 sean-k-mooney dansmith: stephenfin started usign it as part of the nova net removal
15:13:07 sean-k-mooney so yes
15:13:15 stephenfin Yup, it was picked up in one of the "Stop using nova-network objects as containers" patches
15:13:15 dansmith yeah I see
15:13:42 stephenfin mtreinish: That's okay. To be honest, the more crucial fix is the subunit fix and I need lifeless for that
15:13:42 sean-k-mooney so floating ips are not ports and do not normally have port details
15:13:46 dansmith okay just wanted to see what used it and that it was okay with None
15:14:02 sean-k-mooney there is a neutorn extnion to expose the port_details of the prot that they are attached too when attached
15:14:13 sean-k-mooney but its optional
15:14:19 sean-k-mooney apparently
15:14:35 stephenfin mtreinish: https://github.com/testing-cabal/subunit/pull/40 for reference (I think I referenced it the stestr PR)
15:17:31 sean-k-mooney stephenfin: when did we deprecate teh proxy api
15:18:08 dansmith stephenfin: so because this broke the neutron gate, I assume we have tempest tests that actually use this proxy api, right?
15:18:17 stephenfin sean-k-mooney: nova/api/openstack/api_version_request.py tells me 2.36
15:18:50 stephenfin dansmith: correct. the reason we didn't see it was because we test using ML2-OVS or ML2-LB (I don't know which) in the gate, not ML2-OVN
15:18:56 sean-k-mooney well it was only added to neutron in 2018
15:18:57 sean-k-mooney https://github.com/openstack/neutron-lib/commit/24a0877d1d234830a36794388bb342d3d91b9230
15:19:05 sean-k-mooney which is well after we had deprecated it
15:19:17 dansmith stephenfin: yeah I get that. are the tests _testing_ the proxy or using it out of convenience?
15:19:19 sean-k-mooney 2.36 was newton
15:19:48 stephenfin sean-k-mooney: Yup, it's not the proxy API that was wrong, it was how I updated things to stop using nova-network o.vo's
15:20:29 sean-k-mooney stephenfin: what im saying is the proxy api should not conatin that info
15:20:54 sean-k-mooney we deprecated itn in 2016 that extention was only added in 2018
15:21:15 dansmith sean-k-mooney: I'm guessing we had it in there before via other mechanisms
15:21:18 sean-k-mooney so there should be noting relying on it for fip port details
15:21:28 sean-k-mooney ya proably
15:21:34 dansmith sean-k-mooney: looking at the older code I was originally looking at, it was still putting those things in there but in a different way
15:21:38 stephenfin dansmith: That I don't know. Maybe ralonsoh knows since he had a patch up against neutron to disable the broken job yesterday
15:21:53 sean-k-mooney the correct way to look this up before was you get all the ports attach to the vm then get the fip for each port
15:22:05 openstackgerrit Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to servers actions API https://review.opendev.org/706179
15:22:12 ralonsoh stephenfin, yes, we are marking this CI job as non-voting
15:22:13 stephenfin sean-k-mooney: yeah, what dansmith said. We were getting this information by always making a second call to neutron for the ports
15:22:41 stephenfin ralonsoh: Which job is it? dansmith is curious as to what's actually using these deprecated APIs
15:22:49 sean-k-mooney stephenfin: yes which is the correct way to do it if the extention is not available
15:23:24 stephenfin sean-k-mooney: I know. I made a mistake when updating things because I didn't know port_details was optional and figured I should use it
15:23:31 dansmith ralonsoh: job, but specifically which test(s)
15:23:51 ralonsoh dansmith, an example of CI job failing https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_db7/705779/1/check/neutron-ovn-tempest-ovs-release/db7becd/testr_results.html
15:23:54 stephenfin In hindsight, it should have been a separate patch :(
15:24:09 ralonsoh stephenfin, dansmith those ones with exception message
15:24:11 ralonsoh "<class 'nova.exception.PortNotFound'>"
15:24:28 dansmith ralonsoh: okay so some specifically for that api, but also some general ones it looks like
15:24:54 stephenfin dansmith: I assume you're suggesting that the general ones shouldn't be using the proxy APIs?
15:24:55 spatel sean-k-mooney: morning
15:25:09 stephenfin i.e. test_server_basic_ops
15:25:27 sean-k-mooney stephenfin: ya i think the secnairo test shoudl be updated
15:25:40 spatel Could you check this CPU policy for vm - http://paste.openstack.org/show/789225/
15:25:46 dansmith stephenfin: well, maybe, but it's also maybe interesting that if the alternative is very onerous ... that tells us something
15:26:18 sean-k-mooney the other way is not really that hard. its just more api requests
15:26:23 stephenfin dansmith: that everyone is probably ignoring us and using these deprecated APIs anyway? :)
15:26:31 dansmith stephenfin: like, if the alternative is 20 calls to neutron and a bunch of local mapping, then that may be an indication that we kinda messed up
15:26:38 dansmith stephenfin: indeed
15:27:20 stephenfin less users = less bugs = more time for stephenfin to eat popcorn
15:29:15 sean-k-mooney the old way is just neutron port list device-id=<vm uuid> then you loop over that calling /v2.0/floatingips?port_id=<port uuid>)
15:30:25 sean-k-mooney so you get the port detalis form the orginal call to list all the ports on a vm and then map it to the fip from the second call
15:30:47 dansmith yeah, this is exactly what I'm talking about :)
15:30:55 sean-k-mooney if you know the fip you jsut wrok backward first do a fip show get the port and then do a port show
15:31:23 sean-k-mooney port_details on the fip is a proxy api in neutorn to get info form a different resouce
15:31:38 sean-k-mooney the added it as an optional exteion to make it 1 call
15:32:02 sean-k-mooney but each ml2 driver has to advertise support and ovn did not
15:32:18 kashyap stephenfin: Hehe, figured as much. (And I like the expression)
15:43:46 Sundar sean-k-mooney: Re. your message "that was the latest set that were up at that point", it got updated on Jan 10 to remove the specific exception that you posted: https://review.opendev.org/#/c/698846/2/cyborg/common/nova_client.py. Please rebase if your env is older. Sorry for the trouble.
15:46:16 openstackgerrit Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.opendev.org/552924
15:46:26 sean-k-mooney Sundar: i used the local.conf i posted
15:46:52 sean-k-mooney which was pulling in form a new branch
15:47:02 sean-k-mooney ill double check
15:47:36 openstackgerrit Merged openstack/nova master: libvirt: Rename _is_storage_shared_with to _is_path_shared_with https://review.opendev.org/693337
15:48:29 sean-k-mooney strang it seam to have checkout out master...
15:49:00 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Merge unnecessary 'NovaProxyRequestHandlerBase' separation https://review.opendev.org/705655

Earlier   Later