| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-15 | |||
| 16:01:20 | mriedem | ralonsoh: i'd ask moshele and/or cfriesen | |
| 16:01:36 | ralonsoh | mriedem: thanks! | |
| 16:01:53 | ralonsoh | moshele, cfriesen: ping | |
| 16:02:25 | moshele | ralonsoh: hi | |
| 16:02:48 | ralonsoh | moshele: can you take a look at my comment in #link https://bugs.launchpad.net/nova/+bug/1679703? | |
| 16:02:48 | openstack | Launchpad bug 1679703 in OpenStack Compute (nova) "Unable to boot instance with VF( direct-port ) because the PF is not online" [Undecided,New] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) | |
| 16:02:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 16:02:56 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 16:02:58 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 16:03:01 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 16:03:05 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 16:03:06 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 16:03:11 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 16:03:30 | ralonsoh | moshele: it's about attaching a PF to a VM and then binding it again to the OS | |
| 16:03:40 | ralonsoh | moshele: the VF information is lost | |
| 16:04:28 | moshele | ralonsoh: I would expect libvirt to restore the num of vfs | |
| 16:04:53 | moshele | ralonsoh: did you send mail in libvrit community? | |
| 16:04:56 | ralonsoh | moshele: so you think the first option is the valid one | |
| 16:05:10 | ralonsoh | moshele: ahhh, so libvirt itself | |
| 16:05:17 | ralonsoh | moshele: ok, I'll do it | |
| 16:07:42 | moshele | ralonsoh: let me know what they say | |
| 16:18:33 | lyarwood | mdbooth: https://review.openstack.org/#/c/463992/3/nova/tests/unit/virt/libvirt/test_config.py - I'm not missing a volume driver where we are using netfs am I? | |
| 16:19:39 | mdbooth | lyarwood: Bizarrely, I'm currently looking at exactly the same thing | |
| 16:20:52 | lyarwood | haha great | |
| 16:20:55 | mdbooth | lyarwood: We don't appear to have a volume driver which sets netfs | |
| 16:21:15 | mdbooth | Although I'm just looking at all uses of source_protocol | |
| 16:21:26 | mdbooth | Possible it could be imported from somewhere | |
| 16:21:54 | mriedem | mdbooth: yeah, that was added when gluster was added, | |
| 16:22:07 | mriedem | i didn't remove it because i didn't know what might rely on it and break | |
| 16:22:26 | mriedem | in https://review.openstack.org/#/c/463992/ | |
| 16:22:27 | mdbooth | mriedem: I saw that, just looking for myself. | |
| 16:22:30 | lyarwood | yeah, I assumed we were setting it via connection_info in LibvirtNetVolumeDriver but I can't find anything on the cinder side | |
| 16:22:30 | mriedem | ok | |
| 16:22:43 | openstackgerrit | Dan Smith proposed openstack/nova master: Make compute auto-disable itself if builds are failing https://review.openstack.org/463597 | |
| 16:22:57 | mriedem | lyarwood: the source_protocol is set in nova | |
| 16:23:03 | mriedem | in the volume driver's get_config method | |
| 16:23:25 | lyarwood | mriedem: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L46 | |
| 16:23:26 | mriedem | we could potentially remove the netfs stuff after this since it appears to be dead code now | |
| 16:23:27 | mdbooth | nova/virt/libvirt/volume/net.py: source_protocol = connection_info['driver_volume_type'] | |
| 16:24:00 | lyarwood | argh wrong line number | |
| 16:24:08 | mriedem | mdbooth: ok, i think that's just probably wrong | |
| 16:24:12 | mriedem | i.e. bad var name | |
| 16:24:43 | mdbooth | mriedem: It seems to be used in get_config there | |
| 16:24:48 | lyarwood | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L96 | |
| 16:24:55 | mdbooth | Yeah, that one | |
| 16:25:10 | mdbooth | So that means it could come from cinder and end up in the libvirt xml | |
| 16:26:12 | mdbooth | cinder doesn't appear to set driver_volume_type to netfs anywhere, though | |
| 16:26:17 | mriedem | blech | |
| 16:26:35 | mriedem | btw, the rbd and iscsi net stuff crammed into the same driver code is not fun | |
| 16:26:48 | mdbooth | Yep | |
| 16:27:16 | mriedem | there could have been a cinder driver years ago that had netfs | |
| 16:28:05 | openstackgerrit | Moshe Levi proposed openstack/nova master: HW offload support for openvswitch https://review.openstack.org/398265 | |
| 16:28:39 | mriedem | i checked grizzly, icehouse and kilo though and found nothing | |
| 16:30:52 | dansmith | sdague: check it out: http://logs.openstack.org/37/458537/19/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e290ec2/logs/screen-n-api.txt.gz?level=TRACE#_May_11_20_08_20_211256 | |
| 16:31:13 | dansmith | I assume that's the mysql server complaining | |
| 16:33:14 | sdague | yeh, probably | |
| 16:33:31 | sdague | I'm pretty sure when you see that it's coming back from the server | |
| 16:33:38 | sdague | this the superconductor fleet? | |
| 16:33:45 | dansmith | sdague: can I raise that in devstack easily or do I need to make something new? | |
| 16:33:52 | dansmith | sdague: yeah but single node | |
| 16:34:00 | sdague | you can change it in devstack, let me find the line | |
| 16:34:04 | dansmith | sdague: do you know if I can control the size of our connection pool? | |
| 16:34:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI._find_service https://review.openstack.org/461519 | |
| 16:34:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle uuid in HostAPI.compute_node_get https://review.openstack.org/461532 | |
| 16:34:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use uuid for id in Services API https://review.openstack.org/464280 | |
| 16:34:26 | mriedem | dansmith: you can | |
| 16:34:38 | mriedem | we had to do that for postgres at some point | |
| 16:34:44 | sdague | https://github.com/openstack-dev/devstack/blob/27df725179045a2e91a34eb91e29e52d0fe5f663/lib/databases/mysql#L99 .... | |
| 16:34:48 | mriedem | https://github.com/openstack-dev/devstack/tree/master/lib/databases | |
| 16:34:51 | sdague | we're exceeding 1024? | |
| 16:34:59 | sdague | that seems suspicious to me | |
| 16:35:27 | dansmith | well, what's our connection pool size? | |
| 16:36:26 | mriedem | we used to save off the mysql config i thought | |
| 16:36:28 | mriedem | but i can't find it | |
| 16:36:41 | dansmith | connection pool size would be something in our own config | |
| 16:37:06 | dansmith | just noticed by the way, | |
| 16:37:13 | dansmith | that we don't get the config dump at the beginning of n-api logging anymore | |
| 16:37:18 | sdague | dansmith: I don't know that we're setting the pool | |
| 16:37:19 | dansmith | maybe because of the wsgi patch? | |
| 16:38:46 | dansmith | looks like 5 per worker maybe | |
| 16:38:49 | dansmith | which is pretty low | |
| 16:39:06 | dansmith | that would be ~204 workers to exhaust 1024 | |
| 16:39:11 | sdague | dansmith: I still see the dump | |
| 16:39:15 | sdague | http://logs.openstack.org/26/464326/1/check/gate-tempest-dsvm-neutron-src-oslo.context-ubuntu-xenial/4fc411c/logs/screen-n-api.txt.gz#_May_14_11_27_57_370073 | |
| 16:39:18 | dansmith | which seems unlikely on a single node | |
| 16:39:21 | sdague | it's just a little later | |
| 16:39:53 | dansmith | oh okay, that seems odd given we're loading extensions before that, but okay | |
| 16:40:25 | sdague | I think the new route system compiles early | |
| 16:40:33 | dansmith | okay | |
| 16:40:35 | sdague | the stuff that completely does away with the extension loader | |
| 16:40:53 | sdague | we honestly could probably turn off debug on those | |
| 16:41:11 | dansmith | now that th3y | |
| 16:41:16 | dansmith | they are all implicit yeah | |
| 16:41:25 | dansmith | anyway, I dunno what to do about this connection thing | |
| 16:42:14 | dansmith | presumably if we go higher we also need to set maxfds | |
| 16:42:20 | dansmith | unless we already have for something else | |
| 16:43:02 | sdague | yeh, I don't know, but I was pretty sure that was the db rejecting things. Any chance we're leaking db connections or something? | |
| 16:44:04 | dansmith | nothing that fundamental has changed recently, so I dunno | |
| 16:46:29 | dansmith | looks like we may be limited below max_connections due to ulimit on descriptors anyway | |
| 16:46:34 | dansmith | so maybe we're not getting 1024 even | |